A
TALE OF TWO LOG FILES PART TWO
Tiger Shark
from Antionline has kindly given his permission for his tutorial to be
hosted at The Taz.
You can find the original post here:
http://www.antionline.com/showthread.php?s=&threadid=258977
Enjoy
I really apologize for the long period between "episodes". It was due
to two factors:-
1. Radisson Hotels and Orlando International Airport in Florida are
somewhat retarded. No wireless at the airport, and a choice between
internet over the TV or 89 cents a minute to dialup the internet from
the laptop. Neither of which was accepable or useful.....
2. My sweetie and my friends kept me sufficiently busy to make a quick
wardrive impossible. When I wrote my note last night I was at a friends
house for a BBQ, was a tad drunk and was being yelled at by my other
sweetie, (our hostess), for working on the computer when I only see her
once a year......
So.... To Part II
Title: A Tale of Two Log Files (Part II)
Subtitle: How Proper Procedure and Comprehensive Logging make an
Administrators job easier.
This is a story. It's fictional and not necessarily
factually/technically correct in all cases but I am using it to
demonstrate two things that are very important to an administrator in
the event of a compromise, the procedure and the logs. Both go hand in
hand in the event of a compromise and both must be in place prior to
the event itself. The proper planning prior to the event will speed up
the investigation and save time and therefore money in the "clean-up"
and mitigation of the breach. It's a lot to do with the mindset and
approach whether you are prepared or not and that is what I will try to
show during the story.
Throughout the story you will find numbers in []. They point to the
notes at the end of the story. The notes are meant to show what the
participants did right or wrong, what should have been done prior to
the event or what could have been done better.
This one was written with no technical reference available.... on a
plane..... just ignore the inaccuracies.....
--------------------------------------------------------------------------------------------------
Following his procedure Gary had carefully carried out the remote
portion of his investigation of Mike Panoff's computer and was somewhat
surprised to see that everything was as it should be. He called network
administration and requested a new computer imaged for Mike Panoff's
department be brought to Mike's office, picked up his forensic CD
carrying his "toolkit" and made his way to Mike's office.
"What's up then Gary", Mike said as he entered.
"Ahhh... Not much... There's nothing wrong with your computer from my
point of view but I noticed the CPU fan seems to be failing. I've
ordered you a new box and it'll be here in the hour. I'm doing the
netadmins a favor and running the diagnostics for them"
"But what was it that made you look at it in the first place?"
"Oh, that...." Gary laughed, "Dumbass here misread a log file and
transposed a couple of numbers. Those logs turn you cross-eyed after
the first ten minutes" [8]
Gary and Mike carried on chatting as Gary mapped a network drive and
began running through his repertoire of tricks sending the output away
to the newly mapped drive. He didn't have to look to his procedure book
to determine what he should be doing next which may have raised Mike's
suspicion's because he had built the CD from the procedure manual and
had practiced with it. He had batch files written for the more complex
switch requirements of some tools so that he wouldn't have to remember
the commands or refer to a manual for just this situation. All he
needed to know was that he needed to run every tool on the CD, with the
batch file if it was available and the location to send the resulting
data. [9]
An hour and fourty seven minutes after leaving, Dirk was again parking
his car in TFCU's parking lot. His spirits were up from the initial low
of the day and he had promised himself to have a "cold one" for Fyodor
next time he had the chance. The NMap scans he ran against his sixteen
IP public subnet had shown that there was no direct exposure to the
public network. Small victory, but a victory all the same. He dropped
into the CEO's office on the way back to his own.
"Well, I have some news.... It wasn't directly from the outside. I
can't find a way in from the outside"
"Er... But that's not good news then. Doesn't that mean that "Al" is an
employee?"
"It could, but that is yet to be seen"
"I'm not sure I understand.... If it isn't from outside then it must be
from inside. Who's computer did the email come from?"
"I don't know right now, I haven't checked. But I'm telling you that
right now we don't want to be accusing anyone of anything".
"I'd like to know who owns that computer.... and I'd like to know
pretty soon".
"Mike, listen to me. There are a lot of ways that someone within the
network messed up a machine without knowing it and now someone outside
controls their computer." [10]
"You just told me that there was no access from the outside, so that's
not possible"
"Mike, it is.... It's like.... er... ET calling home.... I don't
remember the proper term but I read about it a few months ago. It's a
way around the firewalls" [11]
"So you are saying that even though no-one can get in from outside, the
firewall we payed that contractor almost $3000 for a few years back is
useless?"
"No, no... not at all. I have to let some traffic flow freely from the
inside to the out or you won't get your email, other staff won't be
able to use the web etc. etc.. So if Al got a program inside here that
calls home through the... um, web interface that I have to have open
then there isn't a damned thing I can do to stop it." [12]
Gary leant back in his chair as he addressed the meeting. The entire
team was assembled and he had outlined the initial evidence that
indicated a compromise had taken place.
"I just completed the data collection from a user's computer that may
have been used to steal the data. The computer has been switched off
and removed from the user's office and he has been given a replacement.
I took a quick look at the data I collected and there is one thing that
jumps out at me that seems a little odd but I need to get the drive
imaged and secured before I can look deeper into that."
"Can you be more clear" said Bill Steel, the legal representative on
the team, "I mean do we have a suspect or are there other possibilities"
"Well, at this point I'm not going to point a finger anywhere. The odd
thing on that box is a scheduled backup for 6:00pm nightly. It's odd
because by policy all user data is forced up to the file servers to
keep it safe so there is no reason to run backups on the client
workstations. However, until I can access a copy of the drive I can't
determine what is really there and why.
The meeting continued as the parties moved, step by step, through the
series of questions they need to have answers to so that they can
determine the course of action the company was going to take in this
instance. The sticking point was "Al". Having not had subsequent
contact with him the team had no way of knowing what he was going to
request. Clearly it was money but to a banking institution the amount
would be of great interest. The meeting adjourned with no initial
recommendation which would be postponed until after "Al" had made his
demands clear. This bothered Gary somewhat in that until the
recommendations were formulated and accepted by the board he had to
treat everything as if it would have to be presented in a court of law
and this would slow him down. He'd made that point at the meeting to
ensure everyone was aware. [13]
Dirk sat staring blankly at his monitor. "Oh dear.... Why did it have
to be her box, why couldn't it have been that moron in Sales? I
wouldn't mind chasing him down and getting his ass nailed" he thought.
After leaving Mike's office he had decided to start by looking at the
computer that sent the email last night. Having checked his records the
IP and DNS name of the machine indicated it was Amy's computer. [14]
His mind gyrated as he tried to determine how to approach her and what
to do when he got there when suddenly he thought, "Hah, it can't be
Amy..... Yes!.... She has no access the the database..... perfect...and
I can prove it.... The server's log files will prove that." He scoured
the log files for the previous week searching for anything that would
point to Amy accessing the server. He was disappointed until he saw the
time of the access, 10:27pm six days ago. There wasn't a successful
login but there had been two attempts. "Two tries" he thought, "Someone
didn't want to trip the automatic lockout. It couldn't have been her,
she would have used her password". Poking around a little more he came
to the transaction logs for that date. Scrolling down through the
endless text he noticed some odd entries. "What on earth is that?" he
thought, "Never seen entries like that..."
SELECT username FROM users WHERE MID$(username, 3) = "m" AND 1 = 1;
There were lots of them, rotating the numbers and the letters in the
"where" statement. After looking at the entries for a while he thought
he could see what was going on. "Someone is doing something to the
administrator name in the user tables", he thought, "But what's the
'AND 1=1' all about?" He was familiar with basic SQL queries but his
practical knowledge was limited. Looking further down through the log
he could see where the queries that extracted account information from
the customer tables. "Well, for right now I can see where the leak took
place but where the queries originated from is anyone's guess. I'll
make backups of these logs and then see if I can find where the queries
came from." he thought. [15]
Gary moved across to his Secured Logging Systems analysis console while
the image of Mike Panoff's drive was being completed, documented and
secured. "No point in sitting on my hands", he thought, "If I'm lucky
the internal IDS logs might show something of interest on Mike's box".
He filtered the output to show only alerts from the internal sensors
and started looking through the logs. While it isn't unusual for
workstations to trigger portscan alerts one caught Gary's eye.
Portscan from 192.168.70.153 Ended: Time: 12 seconds, Hosts: 18, TCP:
18, UDP: 0
"What's the Cincinnatti office doing kicking off a portscan? It must
have only just exceeded the threshold", he thought as he opened the
portscan log itself. "The time is right and if Mike's box is one of the
target boxes then things may be coming together". He looked down to the
appropriate time and date and found the offending IP address. There it
was, Mike's IP address had been scanned from Cincinnatti about 30
minutes before the login to the server took place. "Something isn't
right... " he thought, "the scan is against port 80, Mike's box didn't
have 80 open. I'm going to need that image to follow this track. First
let's see who our new "potential perp from Cincy' might be, maybe we
have ourselves a winner." he thought reaching for the phone. His hand
didn't get to it before it rang. He picked it up expecting to hear that
the techs were done with the drive imaging procedure and that he could
get it back.
"Hello, Gary speaking"
"Tell your board that the price is ten thousand dollars. I'm not a
greedy lad and ten grand isn't much to your bosses, they could probably
each pay it from their own pockets and not miss it".
Gary grabbed for a pencil after the initial shock of hearing "Al's"
voice again. He needed to get the converstion down verbatim if he could
but he already knew that he wouldn't be able to do that. "What I can
scribble down is what they are going to have to get" he thought as he
checked his watch and noted down the time.
"Ten grand is a nice chunk of change, I dunno what they are going to
say about that" Gary replied
"Bullshit, and you know it. The information I have here is worth way
more than ten grand.... Tell them to be smart.... and keep your eye on
your email"...
The click of the phone found Gary still writing frantically to try to
document the converstion precisely. He'd done a pretty good job, "a
word here a word there" he thought as he finished up and reached for
the phone again. "We need another IRT meeting, and fast" he thought as
he began dialing familiar numbers
-----------------------------------------------------------------------------------------------------
The Notes:-
[8] Here's Gary lying again, but it's all in a good cause. He's come up
with a story for why the box needs replacement and why it came to his
attention in the first place. They don't have to even be good stories
for most users because they trust that you know what you are doing.
Admitting to making a mistake goes a long way to having people believe
you. It doesn't matter that you didn't make a mistake, the user is
likely to empathize and accept your story more readily.
[9] This is one of the great benefits of being prepared. Gary doesn't
have to concentrate so hard on what he is doing to the exclusion of all
else. His tools are all there, organized, easy to use and they follow
the policy laid down. If the policy requires that certain tools be run
in a certain order place them in folders on the CD called "1st", "2nd"
etc. The more you can do beforehand to make your task easy in the real
event then the less stressed and more effective you will be. A further
side benefit is demonstrated in the story, Gary can go about his
difficult task while making it seem easy, stress free and routine
without raising people's suspicions. Furthermore, he can complete any
documentation of the tools run and in what order from the time/date
stamps on the files he is creating.
[10] Kudos again to Dirk. His boss is hell bent on finding a culprit
quickly. He's made a decision, based on a lack of information, and he
now wants action. Dirk has, quite rightly, pointed out that there are
alternatives that are equally probable and that rash action should not
be taken. He also didn't accuse anyone of doing something maliciously,
he used the term "messed up" that implys an accident rather than using
a more purjorative term such as "downloaded something" which implies a
more deliberate act. This helps to keep the stress level of the major
stakeholder's down a little and can keep them from interfering in the
wrong way.
[11] You don't have to be a cracker to be effective in security. But
you do need to keep up with what a cracker can and can't do. You don't
even need to remember the details, just that it can be done. Without
that knowledge it would have been easy for Dirk to also conclude that
this is an inside job. Keeping up with the knowledge means you don't
have to discover things for yourself. It also allows you to be creative
in your thinking, as a cracker would, in the ways these weaknesses can
be exploited.
[12] Dirk thought about how he would phrase technical details so as not
to complicate the issue with jargon unless it is absolutely necessary,
(which is usually only at the conclusion of an investigation). This
just isn't the time to be throwing around port numbers and protocol
names to stressed executives that have no idea what you are talking
about. The odds are they will ask for clarification which costs you
more time and there is a high probability that they will inadvertently
misrepresent what you said to others thus confusing the issue and
starting the rumor mill turning.
[13] Gary is on top of things as usual. It is critically important that
the IRT understand that the investigation must proceed from the
beginning at a "litigeous" pace, meaning that the technicians are
expected to do everything as if it will be presented in a court of law.
This is much slower than moving at an "investigative" pace which lacks
much of the documentation and evidence preservation required by a court
of law. It is also of critical importance to ensure that the board
understands that when you will move from the higher requirement to the
lower the evidence will be tainted. It must be stressed and understood
by both the IRT and the board of directors that the two methodologies
are exclusive and once the litigious methodology is departed from there
is no going back with any hope of a successful prosecution.
[14] Regardless of the fact that there are people you like, people you
dislike, people you trust and people you don't there is one single rule
in an investigation that you must follow. You distrust and dislike them
all equally, it's as simple as that. If you don't you will allow
preconceptions to cloud the process and possibly, unwittingly, allow
them to move you away from the "truth".
[15] Dirk's lack of preparation and knowledge is beginning to show
through. He's "playing" with a log file that may contain evidence but
he isn't documenting it and he didn't make copies before he looked at
them. If Mike choses to try to prosecute the perpetrator in the future
the chances are high that he will never get the case to court. Now he's
found his evidence he is making the appropriate copies, but this is too
late. The copies should be made first, preferably to "write once" media
such as CD-R if a litigious course is to be taken and then they should
be searched and manipulated from the read only copy of the backup.
Original Tutorial
Submitted by nokia for TheTAZZone-TAZForum
Originally posted on March 4th, 2006 here
Do not use, republish, in whole or in part, without the consent of
the Author. TheTAZZone policy is that Authors retain the rights to the
work they submit and/or post...we do not sell, publish, transmit, or
have the right to give permission for such...TheTAZZone merely retains
the right to use, retain, and publish submitted work within it's
Network.