A
TALE OF TWO LOG FILES
Tiger Shark
from Antionline has kindly given his permission for his tutorial to be
hosted at The Taz.
You can find the original post here:
http://www.antionline.com/showthread.php?s=&threadid=258775
Enjoy
Subtitle: How Proper Procedure and Comprehensive Logging make an
Administrators job easier.
This is a story. It's fictional and not necessarily
factually/technically correct in all cases but I am using it to
demonstrate two things that are very important to an administrator in
the event of a compromise, the procedure and the logs. Both go hand in
hand in the event of a compromise and both must be in place prior to
the event itself. The proper planning prior to the event will speed up
the investigation and save time and therefore money in the "clean-up"
and mitigation of the breach. It's a lot to do with the mindset and
approach whether you are prepared or not and that is what I will try to
show during the story.
Throughout the story you will find numbers in []. They point to the
notes at the end of the story. The notes are meant to show what the
participants did right or wrong, what should have been done prior to
the event or what could have been done better.
-----------------------------------------------------------------------------------------------------------------
Nestled in a small industrial park outside Seattle, Tree Fellers Credit
Union, (TFCU), is a small financial institution of some 150 employees,
Dirk Gardner had recently started there as the company IT manager after
many years of IT functions being outsourced to a series of contractors.
He was still in the process of gathering and understanding the
convoluted implementations and systems the string of contractors had
left him with. Returning to his office, morning coffee in hand, he
almost bumped into Amy Walker loaded down with yesterday's mail, they
exchanged pleasantries and went about their business. Amy was the
receptionist at TFCU's headquarters facility and had been working for
the company almost since the day it had opened for business, he liked
Amy, after all, she was attractive and single.
Halfway across the country in Kansas City, Gary Cunez drove up to the
central data processing facility of Scales, Kohl & Brenner, an
international investment banking organization with offices throughout
the USA. As he pulled into his parking space Mike Panoff, the personnel
manager of the facility was getting out of his car.
"Whoo hoo, lookout folks, it's the security geek" Mike said in his
usual joking fashion.
"Be afraid", retorted Gary, "Be very afraid.... I'm watching you today"
he laughed.
Both Dirk and Gary were expecting a quiet Friday followed by a lazy
summer weekend. They both had plans that would change.
Dirk began checking his email, sipping his coffee as he went. It was
all pretty usual with the customary a pile of spam, "Damn them" he
thought, "You update the filters one day and the very next day they
find a way around them". One message caught his eye, not only for the
time it was sent but the fact that it was from the CEO. "What was the
CEO doing sending me emails at 2:40am?" he thought. Opening it up he
realized that all was not well at TFCU. It simply read:-
"I'll be contacting you soon..... It will be in your best interest to
cooperate. Wait for a message from Al."
This terse note was followed by a two column list of credit card
numbers and social security numbers. Dirk opened TFCU's financial
application, waited for the connection and authentication to the SQL
server and typed in the first credit card number. "Oh shit" he thought
as the number was found and the social security number for the card
matched the number next to it on the list. After randomly picking three
other numbers from the list and trying them it was all becoming
alarmingly clear. "This is not good" he though as his mind raced
through the myriad of things he should do, and must do, trying to
determine an ordered and potentially "employment-saving" course of
action. "Just to be sure", he thought to himself, "Let's look at the
email headers". His heart sunk as the headers confirmed that the email
had indeed originated from a private address within his own network. [1]
Back in Kansas City, Gary was collecting his voicemail. It was the
usual stuff from the night technicians about occurences on the network
until he reached one with a voice he didn't recognize.
"Hi, you don't know me, just call me Al. I got your number from Active
Directory... You gotta love all the details in there huh? Can't talk
for long but I just want you to know that I'm a little short on cash
and I'm considering taking advantage of some of your customer's
accounts. Before I do maybe we can come to some arrangement.... Check
your email.... Bye now".
Gary listened three times before he determined that he really had heard
what he thought he had heard. Like Dirk he had a sinking feeling that
his job may have just gone on the line and as a first step he saved the
voicemail. Looking through his surprisingly short list of emails in his
inbox he knew which one the voicemail referred to. It was from his own
account at around 4:00am. It said the same as the email Dirk had
received just minutes before in Seattle but had account details that
matched those of current Scales, Kohl & Brenner customers from all
over the world. "Well", Gary thought to himself, "I guess this is where
I find out if I'm worth what I'm paid.", as he reached for the policy
manual he had worked so hard on two years ago and had been testing and
updating regularly ever since. [2]
This time Dirk nearly knocked Amy flat as he rounded the corner near
the CEO's office. Apologizing hurriedly, he burst into the CEO's
office..
"Er, I need to talk you you right now", Dirk said as he nodded
apologetically at the CFO sat across the desk from the CEO.
"Can it wait, this is pretty important"
"No, I'm afraid not. This is very important, it's a security issue"
"Well let's make it quick, I have another meeting in 20 minutes"
Dirk explained the email he had received, how it had come from the CEO
himself and that he had confirmed that the information in it was
genuine and the source of the email was from within the company network.
"So where do we go from here", the CEO asked, "We can't bring
everything down. What are you going to do?"
"Right now..... I don't know", replied Dirk. "I know that doesn't sound
good but I need time to think this through"
"Isn't there a procedure for this?"
"I really don't know", Dirk nervously replied. "The previous
contractors never really did anything with a firm plan in mind so I am
unsure what resources I have to hand just yet. That's part of what I
need to think about".
"Look, if customer information is in this "Al's" hands we need to move
quickly." interjected the CFO.
"What do you suggest?" Dirk retorted
"Well... I don't know, that's your job"
"Yes, and that's what I'm going to try to do and my first task, in the
absence of a procedure, will be to develop one as I go. It's not
something I can rush" [3]
Meanwhile Gary was putting the phone down on his last phone call to the
Incident Response Team, (IRT). He was happy that he had only had to
call one backup person since all the rest were available. The incident
meeting was set for 10:00am so he was happy for the hour he would have
to prepare. He knew that access to the custom produced system that
managed the customer information was unreachable from outside the
firewall. This left him with two initial avenues. Either it's an
"inside job" or the private network is penetrated in some other way. In
either case it would be difficult to pin this one down. He checks his
logs and systems daily and nothing had peeked his curiosity recently so
his gut feeling was to lean towards an "inside job". A feeling he
quickly brushed aside. [4] His starting point was obvious. Since the
data had to have come from the custom application's server at some
point it would be the right place to look first.
Knowing that the information had to have been stolen prior to 4:00am
when he received the email he began by looking in the application log
for the period prior to that. It took but a few minutes to find a
successful login from a person he knew to be a day staff at 8:30pm the
previous day. A quick phone call confirmed what he thought.
"Hello, Mike Panoff"
"Mike, it's Gary, I told you I'd be watching you. What time did you
leave last night?"
"Wow, you weren't joking." Mike laughed
"No, actually, I'm not. What time did you leave?"
"Er... It was about 5:30 I guess. Why?"
"Did you log off and shut down the machine?"
"Er... I logged off but I leave it switched on and just turn off my
monitor. Why, what's happened?"
"Have you been working on it this morning?"
"Yeah, I.. er... checked my mail and that's it... I had a meeting right
after I got in"
"Good. Do me a favor. Leave it exactly as it is. Don't touch it, log
out, turn it off or anything. If your boss wonders why you aren't
working have her call me, ok?"
"Ok, no problem. Sounds serious, what's up?"
"Ahhh, probably nothing. Hard to tell at the moment. I just need to
look around for a minute. Usually this stuff is just me not properly
understanding what my systems are telling me", Gary said. He thought
about faking a laugh but decided he may not carry it off and give the
game away, so he didn't. [5]
Dirk sat in his office trying to decide the appropriate course of
action. Obviously he needed to work out where the attack came from but
without a clear picture of what had been done in the past it seemed
insurmountable. Was the attack from inside or out? "Is the firewall
effective?", "I know the transaction logs for the SQL server are in
order", "Is there anyone inside the company with a "beef" that would
play games?", were all questions that went through his mind. After ten
minutes of fighting the desire to "hit the keyboard" he realized he was
getting nowhere.
"I'm starting from scratch" he thought..... "So where do I start? Let's
start at the beginning.. test the firewall, what does it allow and to
where" [6]
Dirk picks up the phone apprehensively, "This isn't going to go down
well", he thought as he dialed the CEO's extension.
"Mike, I have to go home for an hour. In order to find out if this was
an inside or outside "job". I need to scan the firewall from the
outside. I need to know what the firewall will allow and what it won't.
I've looked at the rules the contractors put in there but they are
confused... Different people have allowed and denied things in the past
that may conflict.... I have to find out what is really allowed and
denied from outside. Until I know that I really can't be sure where I
need to look next."
"Dirk, what do you mean by that?"
"Well, I'm trying to narrow down the possibilities, I'm trying to
eliminate the easy ones first. If I can eliminate some things quickly
then I can concentrate on those things that are more probable. In that
way I might be able to conclude this more quickly and give you the
answers you need"
"Dirk, I really don't like the idea of you leaving right now, this is a
very big problem and I need my IT staff here to help me."
"Mike, you need to trust me. I can't easily, properly and accurately
scan your network from the inside. I need to be outside, my home is
fifteen minutes away and I can do the job properly from there. The cost
in time is small compared to me trying to guess at this point."
"Ok, make sure your cellphone is on."
"No problem, you have my number, right?"
"Yes"
"OK, I'll be back in an hour or two" [7]
Part 2 is in the works..... Don't you just love flights taking you to
vacations..... Hopefully there'll be a network I can access when I get
there....
-----------------------------------------------------------------------------------------------------
The Notes:-
[1] Dirk made his first good decision, he confirmed that the
compromised information was, in fact, genuine. Sometimes determining
whether or not there is cause for concern is much more difficult and
must be done carefully so as not to disturb or lose any information
that might be critical later on in the investigation. That said it is
important to confirm that there actually has been a compromise. The
procedures and actions you carry out during the investigation are time
consuming and entails involving a lot of people with limited computer
knowledge. At a minimum it will cost money in time and effort and it
could cause a lot of harm to an organizations reputation to be telling
people outside the organization of a compromise and having to rescind
it later.
[2] Discovering a breach in your security is a stressful event that
causes a rush of feelings and thoughts. The combination of feelings and
thoughts is a bad thing because mistakes can easily be made. Accepting
the fact "up front" that there will come a day that, one way or
another, a compromise will take place on a network you are responsible
for allows you to plan thoroughly. The absence of worry over your
employment status, the unhurried ability to ask "what if's" and the
time to test and practice with the tools you will be using is
invaluable. In our story Dirk had been left with the traditional mess a
string of contractors leave and a lack of documentation or policies. He
is "flying by the seat of his pants". Gary, on the other hand, has a
network that he has implemented the security mechanisms for himself,
tested them and finally written/tested the policies for.
[3] Dirk is on a roll. Despite pressure from the top two executives in
his company he understands that he is in a "bad place" that will only
get worse if he doesn't think carefully about his actions before he
carries them out. In any investigation you will come across something
that may not have been planned for in your procedures. In order to
properly deal with it you need to determine a course of action that is
appropriate, that fit's in with the procedures already in place and
that you document as you go in order to be able to place it as an
addendum to the policy for future reference, you need to "pause". This
takes time and thought. If the CEO comes back and demands legal action
against the attacker then these actions must be sound enough to stand
up in court. It's an old adage but it works - Stop, Think, Act. Better
to spend an hour thinking about the possible repercussions of your
actions than a week trying to find your way out of the mess you made by
them.
[4] It is a human tendency to have a "gut" feeling about something and
to seek evidence to prove that it is so. This is the wrong way to
conduct any investigation. An investigation seeks the truth rather than
evidence that might uphold a theory. All avenues must be looked at.
Gary has made a subconcious decision to disregard a direct attack
through the firewall in preference of an internal attack or one that
originates outside the firewall using an internal resource as the
"stepping off point". While he has set aside the potential for a direct
attack he has not succumbed to the "gut feeling". He has prioritized
his investigation but may still go back and investigate a direct
attack. There is a very fine line between the "prioritization" and the
"gut feeling" but it is one that should be considered while determining
where to start. You must maintain "the big picture" and remember that
even though you are following a course of action that currently
excludes the possibility that something else occurred you absolutely
must not do anything that would preclude you from discovering the
evidence that lies in those things you have discarded as improbable or
impossible.
[5] Your investigation is going to raise the curiosity of everyone you
"touch". Frankly, it's an exciting thing for them and when they are
excited they are going to talk about it, to anyone and everyone. They
don't have any facts and they probably have no idea about what they are
talking about or the possible repercussions of telling people what they
"know". They may also not be particularly discriminatory about who they
tell which could make the problem even bigger, (they may be telling an
inside attacker that s/he has raised the suspicions of the IT staff).
The rule is: Tell nobody outside the IRT what is going on. You may have
to bring people "into the loop" as time goes on but that should be done
formally and with the permission if the IRT itself. Keep it all simple.
You "don't know", "it's just a normal audit of systems"..... Use your
imagination but have the answers to such questions formulated before
you call people to request information. Yes, it takes more time, but in
the end it is the best policy.
[6] It's very easy to "hit the keyboard" and start your investigation
without a policy in place. It's equally easy to find something that
goes outside the parameters set down in your existing policy and carry
on regardless on that keyboard. Don't, it's a bad idea that will come
back to haunt you. TFCU is lucky, Dirk is smart. He thinks before he
acts. He starts with things that will move the investigation forward
quickly, (narrow down the potential issues), without detrimentally
affecting any data, systems or, more importantly, his investigation.
[7] Having thought carefully about his course of action Dirk is now
leaving his chosen trade. He is, unwittingly, entering "people
management". The company's management will be equally, if not more
stressed out than you are. It's their money and reputation you are
dealing with now. You know what you need to do. You probably have an
idea of how much time it might take, but you lied, (let's be honest,
lying is ok sometimes). You started the conversation with a set
paramater, (an hour). By the end of the conversation you have exceeded
the original expectation, (an hour or two), and have given yourself the
time to do what you need to do. During the conversation you
demonstrated the benefits to your course of action and you delineated
the problems to him/her if you don't follow your course. You "bought"
the acceptance of your course of action then you "gave yourself a
chance". It's not necessarily your strength, but it's a fine skill to
learn. It won't be as quick and easy as the story suggests but if it is
something that gives you the "comfort level" to make decisions and move
forward then it is something you need to learn to "manipulate".
Original Tutorial
Submitted by nokia for TheTAZZone-TAZForum
Originally posted on March 4th, 2006 here
Do not use, republish, in whole or in part, without the consent of
the Author. TheTAZZone policy is that Authors retain the rights to the
work they submit and/or post...we do not sell, publish, transmit, or
have the right to give permission for such...TheTAZZone merely retains
the right to use, retain, and publish submitted work within it's
Network.