A
TALE OF TWO LOG FILES PART THREE
Tiger Shark
from Antionline has kindly given his permission for his tutorial to be
hosted at The Taz.
You can find the original post here:
http://www.antionline.com/showthread.php?s=&threadid=259017
Enjoy
Subtitle: How Proper Procedure and Comprehensive Logging make an
Administrators job easier.
This is a story. It's fictional and not necessarily
factually/technically correct in all cases but I am using it to
demonstrate two things that are very important to an administrator, the
procedure and the logs. Both go hand in hand in the event of a
compromise and both must be in place prior to the event itself. The
proper planning prior to the event will speed up the investigation and
save time and therefore money in the "clean-up" and mitigation of the
breach.
Throughout the story you will find numbers in []. They point to the
notes at the end of the story. The notes are meant to show what the
participants did right or wrong, what should have been done prior to
the event or what could have been done better.
-----------------------------------------------------------------------------------------------------------------
"It wasn't a voice I recognized" Mike said to Dirk who was standing
nervously in the doorway.
"You mean you don't think it's an employee then?"
"I know practically everyonehere and the voice just doesn't fit, no-one
has an accent like that here"
"Well, then it does point to a "call home" program then. I looked at
where the email came from and it was Amy's."
"Amy? She's been here forever."
"Well like you just said it doesn't seem like an inside job and Amy
doesn't sound like a man either"
"Oh, yes, you're right. So he want's ten thousand dollars or he starts
using the accounts."
"Well we can't close all the accounts and issue new one's. That would
tell everyone we were compromised"
"Can you find this 'Al'? I saw a program on TV once about how a spy was
caught. Can we do that?"
"I don't think so. I need to look at Amy's computer but finding the
information I would need is going to be difficult. I could really use
some log files but I took a quick look around and where there are any
logs they overwrite themselves when they fill up which sometimes seems
to be overnight on some machines." [16]
"So you're saying that without logs you may never know where all this
came from?"
"Basically, yes. They are one of those things that always seem to get
the low priority. I'm guessing the previous contractors had better
things to do than to bother with them."
"So we don't have a lot of choice then? We are going to have to pay"
Mike said glumly.
"Give me some time, maybe I'll get lucky. Maybe Al is a little sloppy.
When does he want the money?"
"I don't know, he said he'd contact me soon"
"Let me look around. Call me when you know more, ok?"
"OK, I have some calls to make, get back to me if you find anything"
"Will do."
Gary asked Bill Steel to call an emergency board meeting for 2:00pm
today. He needed the 'ok' from the board to move to "investigative"
pace. He had explained to the IRT that while the probability was high
that he would find the outside locations that the attack had been
initiated from the chances are that they were zombies being used by Al
to mask his true location and identity. The recommendation would be
that the company pay Al and move on. It irked Gary a little but he knew
that the chances of successfully finding and being able to prosecute
were minimal and that calling in the FBI would affect the company badly
since the theft would become public once they were called in. [17]
Arriving back at his office Gary decided to take a look and see who's
computer in Cincinnatti was carrying out the scan. A quick check of his
DHCP logs indicated that on the day and time of the scan the computer
belonged to the manager of the Cincinnatti office, Dan Ereg.
"Interesting." thought Gary not doubting that Dan wasn't Al but
beginning to wonder how pervasive this attack was. "I do hope he hasn't
been hopping from box to box for weeks," he thought, "that's going to
be a real pain not to mention the fact that I'm going to have to
explain why I didn't notice it". [18]
Dirk sat forlornly in front of Amy's computer. He'd opened the event
logs to find nothing. "This really sucks," he thought, "those
contractors are useless.... they didn't even turn on auditing. Jesus,
all they had to do was set the domain policy.... Less profit in doing
it right I guess". Amy's voice brought him back from his angry thoughts.
"You don't look very pleased" she said, "Is it something I've done?"
"No, I don't think so. I'm peeved at the old contractors. I'm trying to
find something that should be easy had they done their job
correctly.... But they didn't".
"What are you looking for, maybe I can help"
Dirk laughed, "I don't think you could help me, I don't know what it is
I'm looking for myself".
"That doesn't make much sense," Amy replied, "How do you even know you
should be looking for something then?"
"Ahhh... It's a long story. Maybe I can tell you some other time, when
I get this all cleared up".
"Oh please, you've certainly got my curiosity up" she smiled. The
double meaning went right over Dirk's head.......
Dirk went back to looking around her computer. He was lost and he knew
it. "This is bloody silly," he thought, "Thirty thousand files or more
and no idea where to start. If I'm not careful it's going to look like
Amy did it and I can't believe that." He wandered aimlessly around
Amy's computer for another twenty five minutes and was ready to give up
when he opened the task manager. Looking slowly down the list it struck
him as odd that mstask.exe was running. "What's this?" he thought, "The
scheduler is running". he opened the task scheduler and there was a
single entry named "Daily Backup" set to start at 6:00pm daily. "Hmmm,
it ran last night at 6:00 and it is due to run again at 6:00. What's it
doing?". He quickly checked the properties. What he found made him sit
up suddenly. The task was scheduled to run a program from the system32
folder called "msbackup.exe". "Got it." he thought, "Microsoft's backup
program is "ntbackup.exe". This must be a trojan. There's an IP address
amongst all the switches. I wonder where that is?". He quickly made a
note of the address and returned to his own workstation. He "pinged"
the IP address and got a response. "Cool," he thought, "It's up. Now
what?" [19]
Gary looked up as the technician knocked on his door carrying a copy of
Mike Panoff's imaged drive and a manifest detailing what had occurred
and where the other drives were currently located. He thanked the
technician, filed the manifest after checking it to make sure things
were in order and placed the drive in his machine. He went straight to
the scheduled task which he found to be called "Daily Backup". "No
reason for that whatsoever" he thought as he looked to see what the
executable was. "Now that's an interesting command line", he thought.
"msbackup -l 80 -e cmd.exe..... that looks all too like Netcat...
running as system too because of the scheduler, ouch, nice one." He
made the appropriate notes and moved to the event logs. There he found
new processes being started just after the portscan had taken place.
"VNC was run not long before the access to the financial database.", he
thought, "OK, so he's even making himself a desktop to work from. Lets
take a look at the VNC executable". He opened Explorer and went to the
path shown in the event log. Nothing. "Ok, let's try the recycle bin?"
he thought as he switched to it and found it immediately. "Great, the
deleted time was not long after the data was stolen." He further noted
that there were a couple instances of the VNC executable over the
previous few days. "Hmm... He downloads a new version each time. I hope
he only uses Dan's machine as his first stop internally". He sat back
to think for a minute. "Why was he portscanning for port 80? He has to
know he has Netcat on Mike's box....". It took a full three minutes
before the possible answer came to him. "He can't guarantee which
computers will be left on after work..... Shit, he has several
machines..... I bet there are 18 machines total..... I need that
portscan log again...." [20] He called the network admin and asked that
all 18 machines on the list be checked for the scheduled job and was
surprised when, just 45 minutes later, the admin was standing at his
door with log files of his technicians activity on each machine as Gary
had requested.
"You were spot on.", he said, "Every box has the same scheduled job.
Busy little beaver that hacker friend of yours eh?"
"Jeez.... He wanted to guarantee access didn't he?"
"So what do you want me to do. I can't pull all the boxes, I don't have
replacements."
"Yeah, I know.... tough one.... The board meeting is in 5 minutes so I
can't say right now. We've recommended no action so we can move faster
but the board has to ratify that. Until then I'm supposed to work as if
we were going to court with this stuff. Then again, if we have good
evidence on Mike's box and we pick up the box in Cincy, we should be
good in a court, anything else we have should just be additional nails
in the coffin so just documenting the other machines should be good.
It's something I'd have to check with the legal beagles if the board
wants us to go after him but we should know in an hour or so. Let's
just wait till then."
"Ok, he's a pretty bad hacker though isn't he? He's leaving clues all
over the network."
"Yeah, I would have expected better, even the log files were intact.
Hopefully he won't be any more careful with the box or boxes he uses to
enter the network"
Dirk was now uncertain where to go from here. He'd found the attacker's
computer in the internet, it was up and running and he was somehow
controlling Amy's machine to do whatever he was doing to get to the
financial data. "I need a plan," he thought "this can't be the first
time something like this has happened, someone has to have seen this
before....." Finally, the "light bulb" came on.... "Google" he thought,
"I'm a genius....". He scrambled over to his workstation, opened IE and
went to Google. He typed "incident response methodology" and clicked
"search". "Oh Hell. 201,000 hits....." He added the word "windows" and
re-searched. "That's better" he thought, "cut that down to nearly
42,000 hits. Shit, I'm going to be reading all night...." Despite the
seemingly overwhelming task Dirk felt a kind of warmth. He wasn't
alone, he wasn't the first victim and there was a lot of information
out there about how he should proceed. "Damn, I probably should have
done this earlier.", he thought. [21]
"You have got to be shitting me", Gary said, "Do they know what that
will do?"
"Look Gary, that's the decision of the board. They want you to go as
far as possible and then they will call the FBI." Bill said.
"You explained that our chances of finding this asshole is close to
zero didn't you?"
"Yeah, but the feeling is that if we give the ten grand he'll be back
anyway. Eventually he'll drop the information somewhere and then we're
screwed regardless. They feel it is better to face the issue head on,
tell the investors about the leak and do everything to minimize the
damage. The PR people are already working on the issue. Sorry, but you
get to stay on the slow track"
"Hell, I have eighteen boxes I have to drag off the network already....
Should we just close up shop now?"
"Eighteen, how?"
"It'll take me too long to explain but he has eighteen boxes on this
network alone he can connect to and control, never mind the other
offices."
"Nice, very nice..... Can we handle that?"
"I dunno, listen, I have a thought, what do you think? I'll just leave
the boxes up, I already took down Mike's but that should be ok, he
expects boxes to be turned off sometimes. If I leave the other boxes
alone and let things happen as they will can we still hold up in a
court? I want to just put a monitor on all of them and see if he comes
back. If we maintain a proper evidence trail on the boxes we have can
we allow the other boxes to be tainted and still win on the off chance
we ever get this shit into a court?"
"If your solid on the evidence trail of the boxes you have right now
and can show his activity from your monitors I'll argue it with any
defense attorney but I need a rock solid place to start and I'll need
good data to back up our decision. Can you provide that for me?"
"I think so, but you're the legal beagle.... Do you trust me is the
question?"
"Er.... Oh, to hell with it.... Go for it... Make sure you are letter
perfect on the procedure with everything you do. I need everything
documented and I need you to be able to show reasoning. You do that and
I'll back you and argue it in any court if we can get it there"
"Thanks, I appreciate that. I'll set up the monitors now, the jobs will
kick off in, um, one hour, twenty three minutes.... I'll be ready"
"Good, and good luck... to us all"
"Yeah, we need some, bye." Gary said as he put the phone down. [22]
"Now this really sucks" he thought. "I really don't like this.... I'd
rather close all this down now....."
-----------------------------------------------------------------------------------------------------
The Notes:-
[16] Dirk is beginning to realize that the log files on a network are
his eyes. When log files don't exist or overwrite themselves when
"full" he is blind. All manner of things could have gone on throughout
his network minutes before the log began overwriting itself and he
would have no way of knowing it.
[17] No matter how good a sleuth you think you are you must realize
that once the trail leaves your network it also leaves your logging
systems. Once this has occurred the trail will most probably "dead end"
fairly quickly. Your task is also to operate in the best interests of
your company rather than follow your desire for revenge against your
"violator". The IRT's recommendation to the board must be realistic and
take into account the various aspects of the whole and their
repercussions.
[18] Whether you have the best logging systems in the world or not you
can't expect to find every little thing. Even on relatively small
networks the traffic volume can be huge and diverse. A good attacker
will try to utilize "normal" traffic patterns to mask his malicious
activity. Accept the fact that network traffic is extremely complex and
unless you packet capture everything on the network you are going to
have holes in your logs that may allow malicious traffic to slip by. Do
your job, if you have thought about this situation previously and tried
to make sure that you log the "right" things you should still be able
to piece a trail together. It may not be complete but it will tell you
were 'Al' has been which makes cleanup a lot easier.
[19] Bad move Dirk. He might as well have called Al and told him he had
found one of his tools. You shouldn't make any direct contact with
remote machines at any time during the investigation until you have
collected all the information you can. Once you have that information
use a dial up connection to another ISP or go and use your home
computer to look at the remote machine. You have to be very careful
wherever you take a "peek" from. Al is in the process of committing a
crime. One that could cost your company a lot of money and more
importantly put him in jail with a nice cell-mate called Bubba. He's
going to be more than a little suspicious about probes against his
machine(s). You really don't want to force his hand and have him
release the information for example when you were trying to keep the
compromise quiet.
[20] While there isn't always a good reason for things appearing in
logs there are often sane ones. Gary nearly bypassed the reason for the
portscan and may have become sidetracked by more "exciting" tasks. It's
important to look at each event that is relevant to the attack and try
to determine why it occurred. Gary got lucky, the thought occurred to
him after the fact and he didn't dismiss the question. Had he done so
he wouldn't have found all the holes in his network. It helps to try
placing yourself in the position of the attacker and work backwards.
Why would I portscan this subnet from Cincinnatti trying to find my own
Netcat? That question may come up with an answer much quicker than "Why
would he be scanning me internally"? The difference is subtle but if
you can place yourself in a position where the attackers "problems"
become your's you may speed up the process.
[21] No matter what is happening. No matter how stressful it might be,
you aren't alone and you aren't the first to have been cracked and have
the company's data and it's reputation on the line. No matter how well
prepared you are you will most probably be ill-prepared for something
that turns up. Let's face it, few of us are full time network security
analysts with daily excursions into the world of forensic
investigation. Most of us are network admins who rotate ten different
hats every day. But that's ok, Google will help. It's a mantra repeated
hourly in the computer security world, "If in doubt, Google it" or
"Google is your friend". Learn it and live it. Again, it is better to
spend time finding out what you _need_ to know to be successful than to
ruin your chances of ever being successful.
[22] Unless it's your company and your money things aren't always going
to go your way. You might understand how low the chances of finding the
attacker are going to be but it is the board's decision as to how they
run the company and what may serve it's interests best. You give the
best advice you can, you are clear about your abilities and your
inabilities and you let them make their decision.... It's their
business and that's why they get the "big bucks". It's important that
they know your weaknesses as well as your strengths. It may seem
foreign to be telling your bosses that you "suck" at something but it
is information they need to know. Suggest that they hire a contractor
to help you in your areas of deficiency. The important thing is that
you reach the truth. Unless they are utter idiots they will appreciate
your proactive approach to _their_ problem and your understanding of
your own deficiencies. It might sound like "falling on your sword" for
the good of the company but if they don't understand and appreciate
your committment I can assure you there are better places for your
talents.
Original Tutorial
Submitted by nokia for TheTAZZone-TAZForum
Originally posted on March 4th, 2006 here
Do not use, republish, in whole or in part, without the consent of
the Author. TheTAZZone policy is that Authors retain the rights to the
work they submit and/or post...we do not sell, publish, transmit, or
have the right to give permission for such...TheTAZZone merely retains
the right to use, retain, and publish submitted work within it's
Network.