A
TALE OF TWO LOG FILES PART FOUR-FINAL
Tiger Shark
from Antionline has kindly given his permission for his tutorial to be
hosted at The Taz.
You can find the original post here:
http://www.antionline.com/showthread.php?s=&threadid=259056
Enjoy
Subtitle: How Proper Procedure and Comprehensive Logging make an
Administrators job easier.
This is a story. It's fictional and not necessarily
factually/technically correct in all cases but I am using it to
demonstrate two things that are very important to an administrator, the
procedure and the logs. Both go hand in hand in the event of a
compromise and both must be in place prior to the event itself. The
proper planning prior to the event will speed up the investigation and
save time and therefore money in the "clean-up" and mitigation of the
breach.
Throughout the story you will find numbers in []. They point to the
notes at the end of the story. The notes are meant to show what the
participants did right or wrong, what should have been done prior to
the event or what could have been done better.
-----------------------------------------------------------------------------------------------------------------
During his mammoth reading task Dirk determined that he had found a
reverse telnet connection by using Netcat to make the connection out
through the firewall on port 80 and then spawning a command prompt that
could be controlled from the remote computer. He had further decided
that what he had seen in the transaction logs was a form of SQL
injection that was used to enumerate the tables and their content until
an administrative login name and password could be returned. From there
he concluded that the rest was trivial. He looked at his watch. It was
now 9:30pm. He walked down to Mike's office and found him still there
on the phone. He waited for several minutes while Mike finished his
call.
"You know, I think we have it. I won't go into the details but I know
how he got in, what he did when he was here and thus how he stole the
information. I also think I have the IP address of his computer. Did
you hear from him again yet?"
"No, and to be honest I don't think we will before tomorrow"
"Then maybe we have time to call the authorities and have them deal
with him."
"You think so? Er.... what does this IP address thing give us?"
"It's the unique address of the computer on the internet that he
carries out the attack from".
Mike paused for some while. "You think we can pull it off?"
"It wouldn't be up to us but we need to give the FBI as much time as we
can. I'd say it's now or never. I have the phone number for the local
office right here."
Mike thought again and finally, resignedly said "No. Sorry, this stays
in-house."
"It's your company Mike..... Your decision"
Dirk was a little angry as he walked back to his office. He wanted a
piece of this thief that had come into his network and made him look
bad. "Well, he's not coming back in." he thought as he diverted over to
Amy's office switched on her computer and deleted the offending files
and the scheduled task. "There, screw you... asshole".
Earlier Gary had set up the monitoring for any internal traffic
directed at port 80 and had sat back to think about how he was going to
lock out the access when he needed to. He knew he had eighteen machines
here and one other in the Cincinnatti office. How would he find out how
many other boxes there were? His mind was wandering and the two thought
processes collided. "Oh Duh...." he thought, "Why not just do an 'Al'?
Scan the subnets for port 80 after six in each time zone. They'll show
up on my monitor but I'll be able to recognize mine by the source
address. Perfect". He was left with one other problem. How to determine
if there were more than one access point. "The firewall logs are going
to show me that." he thought as he picked up the phone to Cincinnatti.
He spoke to the admin there and requested the command line for the
scheduled job on Dan Ereg's machine. A few minutes later the phone rang.
"Gary speaking"
"It's Tim in Cincinnatti. I have your command line."
"I just need the IP address it connects to."
"No IP address, it's pointed at a domain name, al.attacker.com"
"Ok, he probably wants to be able to move, thanks"
After putting the phone down he went to the Secure Logging System and
filtered the previous month's logs for the IP address he had resolved
through nslookup. There they all were. "Damn, this guy sucks. He's so
predictable. Always port 80, always 6:00pm in the time zone and only
three machines. Well that will make the cleanup easier" he thought.
Just to be sure Gary then filtered the logs for the previous month
against the internal IP addresses and looked at the transactions
immediately after six pm each night. He found they all pointed to the
same address, al.attacker.com except one, the very first. "Noooo... It
can't be.", he thought "He can't be this frigging dumb.... Can he?" He
quickly ran a Whois against the IP and found it was a common high speed
ISP. "C'mon now.", he thought, "Let this be my lucky day". He opened
google and entered the IP address. "Oh Baby, twelve hits". They were
mostly abuse reports from different locations. "Sweet, a pattern, this
guy has been practicing from home and got himself noticed before". The
eighth one down really caught his eye. It was an email to a list which,
(as many do), contained the headers. There was an email address too.
Better yet the email address showed as a real name, <Joshua
Albin>. "Well what have we here.... It's 'Al'..." Gary laughed out
loud as he reached for the company phone book. He quickly looked up the
number for the President of the Board and dialled it.
"The President's office, Julie speaking, how may I help you?"
"Er, yeah, hi, it's Gary Cunez, Corporate Computer Security Manager. I
really need to talk to the president, like, er...., right now, is he
available?"
"One moment please" as Julie placed him on hold.
A few moments later he was talking to the President.
"Gary, This is Bill Smoltz, the president, what do you have for me?"
"Sir, I'm pretty sure I have 'Al'. His name is Joshua Albin. With some
assistance from his ISP, HighSpeedAtHome.com, I think we can give the
authorities an address."
"How did you manage that. I thought these people did everything they
could to hide themselves?"
"I won't go into detail but he is pretty sloppy and made a single,
rather large mistake. It took only a couple of minutes of digging which
showed a pattern of abusive behaviour and, more importantly an email
address."
"So what do you want me to do?"
"Let's bring in the FBI. With this amount of information they could
have him in hours and he would be out of our hair. Not to mention the
fact that if he's only asking for ten thousand he's probably running
the same trick against others. We can help ourselves and others."
"It's a bit late Gary. Two hours ago a company statement was Fedex'ed
overnight to all our investors stating that their accounts were frozen
without each transaction being accompanied by a code issued with the
statement and that the company would not deal with blackmailers or
other fraudulent activities now or in the future."
"Sir, that's ok. How do you think the investors would feel if they
received a second statement within twenty four hours telling them to
relax, we caught him? I have him Sir, I know it."
"You do have a point.", Bill hesitated, "Ok, you call the FBI and give
them what you have. I have some more calls to make now. I'll get back
to you"
"Thank you Sir. I really feel good about this"
Early the next morning Agent Hicks sat for two hours with Gary as he
went over the evidence he had collected. Gary finished up by showing
him the first IP address, the ISP, the Google results indicating a
pattern from the IP address and finally the email.
"I think you have him there Gary my man", Agent Hicks smiled, "You are
even luckier than you think. I've worked with HighSpeedAtHome before,
several times. They are great record keepers like yourself. This may
only take a phone call or two to start my little ball rolling." He
grinned broadly.
"Phew, I'm glad. It's a gut feeling that this is the guy but everything
points to the owner of that IP being a bad kid in cyberspace."
"I can't argue with you there. Can I use your phone?"
"Yeah"
Agent Hicks made a couple of calls. "Ok, I know who I need to talk to
now," he said, "Let's see what HighSpeedAtHome have for us" as he
dialed the number he just noted down.
"Yeah, Hi, Marvin Brenner please"
"Yes, I'll hold"
Hicks waited.
"Yeah, Marvin Brenner? Hi, This is Agent Hicks of the FBI, you should
have been informed that I would be calling"
"Good. I need to know if there have been connections between the
following two IP addresses in the last month. Can you do that?.....
Good". Agent Hicks read off Gary's Cincinnatti IP address and that of
Joshua Albin. A minute or so went by and he began listening again.
"Good.... Ok, can you confirm that there were no other connection
attempts to the second address I gave you just before or after that
connection was made". Another long pause ensued.
"Ok, the second IP address had two attempted connections on port 1433
five minutes after the connection in question. Can you look at the
source address of the port 1433 attempt and tell me if this was a
single instance or part of a scan?"
"Ok, so you are saying that three weeks ago today at 6:00pm almost
exactly the first IP address connected to the second on port 80 and
that apart from a confirmed portscan for an SQL server from an
unrelated source which touched the second IP address there were no
other connections in or out for more than an hour.... Great, I want you
to hold the logs because there will be a subpoena for them before you
finish work. One final thing, a yes or no question, is the subscriber's
name Albin....." another pause, "William Albin. Great, I'll be seeing
you soon, Bye.
Agent Hicks hung up and turned to Gary.
"Nice work.... It's 'Al' all right, no question. The dumbass made the
first connection go to his home. I have to go, I'll be in touch"
It was 5:30 pm and Joshua was bored. He'd spent most of the day trying
to work out how he was going to get the money without giving himself
away and still wasn't any better off. Since leaving school two years
before and deciding college wasn't for him he had wandered,
unsuccessfully, from job to job. His current employment was the
graveyard shift cleaning a large hotel's kitchen and he hated that too.
The doorbell rang downstairs. "Screw it,", he thought, "the old man can
get it.... he needs the exercise". He heard the door open and some
muffled conversation.
"Josh, Can you come here a minute" William Albin called up the stairs
"What now dad, I'm busy"
"It's a girl to see you son"
"Er, just a minute"
Agent Hicks nodded thanks to William for not alerting his son. Joshua
appeared round the corner and was confronted by two "suits" and two
badges.
"FBI Joshua. Please be calm, we just need to ask you some questions".
Before Joshua could speak Agent Hicks reminded him of his right to
silence and a lawyer. Joshua's only comment was to his father, "I
wasn't going to take the money Dad, I was just showing them how
vulnerable they are.... Dad...."
Agent Hicks colleague took Joshua to the waiting car while Agent Hicks
went upstairs and removed a computer and a laptop from Joshua's room.
Gary took the phone call from Agent Hicks.
"We have him and his computers. No doubt it was him, he partially
confessed to his father when we arrested him. I'm going to be busy for
a while, I'll get back in touch when I have what I need. Just make sure
that your evidence stays clean, ok"
"Yeah, no problem, it stays where it is.... and thanks"
"No problem, you made our job real easy, bye.
Gary called the President again and informed him of the developments.
The second statement was already prepared and was sent out to all the
investors.
An hour later in Seattle Dirk and Amy were just ordering dinner while
he tries to explain the technicalities of how the attacker was causing
her machine to connect to his and then have him control it. He felt
pretty good about the whole thing. He had found the compromise, worked
out how the attacker was doing it and had even found the attackers
computer. Yep, he was "The Boy" and he was going to leverage that all
he possibly could with Amy tonight.
Mike sat in his office at TFCU. "It's getting late he thought" as he
looked at the clock in the system tray of his computer. He noticed it
change from 5:59 to 6:00pm. What he didn't notice was the flicker of
the drive light under the desk as his computer sent a DNS request for
al.attacker.com followed by a SYN........
-----------------------------------------------------------------------------------------------------
The Synopsis
Subsequent investigation and questioning of Joshua proved that the
initial breach took place through a malicious script disguised as an
enticing spam message sent to select members, (usually high ranking
company officers), that insinuated itself into the local security zone
of the machine and downloaded Netcat disguised as msbackup.exe and
created the scheduled job for 6:00pm each evening to set up the reverse
connection through port 80 on the firewall thus making it hard to
detect and even harder to stop. From there Joshua had used other tools
to elevate his privileges and compromise numerous machines throughout
his "conquered" networks. It was then simply an issue of determining
where the financial information was kept and how to get the required
authentication information or use other methods such as SQL injection
to get the data he required.
-----------------------------------------------------------------------------------------------------
The Lessons
Dirk's a nice kid and not a bad admin, but he was utterly unprepared.
It wasn't his fault. We've all seen it, whether it be a new job or
simply going to help a friend. The network that was never built with a
cohesive plan in mind. It takes months to learn it and maintaining it
at the same time means things simply "go by the wayside". Dirk found
himself in a stressful situation which he had no experience with and,
worse yet, had no plan to try to address the problem. People say that
"information is power" and it is true. Logfiles are information.
Therefore logfiles _are_ power. Even if you are unprepared for a
situation a comprehensive logging system can allow you to "muddle
through" without the nicely documented plan your boss would like to see
that will still probably contain "holes". The holes can be filled by
the information in the logfiles. As hard as Dirk tried his efforts were
stymied by the lack of information. That lack of information coupled
with a lack of understanding of how the attacker works led him to
believe that he had a clean network, that he knew what had occurred and
where the attacker was attacking from. The reality is he was left with
a dirty network, he had no way of knowing that other machines were
compromised, he didn't understand that Joshua wasn't attacking him from
his own computer and he didn't know that Amy's machine was a "jump off"
point inside his own network, (though in this case Joshua did use her
machine to get the data he needed, logs may have shown him a different
picture).
Gary has an advantage over Dirk. His company employs him to do the
security. He's been there for a while and he's built his security
architecture himself. He monitors the network, he logs it heavily and
he wrote his procedures while he had time to research them. He also
watches what the potential attackers can do and adjusts his systems
accordingly. So when the "bad thing" happens he can move in, confident
that unless the attack is extraordinary he has a good chance of piecing
it all together. Gary ended up with a clean network, sufficient
information to put Joshua in jail for three years and a handsome pay
raise, (ok, I lied a little ).
There is no reason why the "average" admin can't accomplish what Gary
did. It starts with looking at each part of the network and each
project you take on from a security standpoint. Simple questions like
"Can I log these transactions somewhere?", "How could this be
exploited?", "Can I know who connects to this and when", at the start
and both implementing and documenting everything you can goes a long
way to helping you when the "proverbial" hits the fan. Computers are
cheap and drive space is cheaper. An old PC with a nice big 80 Gig
drive is "chump change" today. Couple that with a CD writer to archive
the logs to at less than $0.50 per CD and your ability to log your
system properly is acheived in less than $2-300. What's the problem?
"Google it" is a mantra amongst security aware admins, and it's a good
one. I would propose another mantra. "If it communicates, log it". I
wouldn't want to be blind in my real life, why would I want to be blind
in my cyber life?
Finally, ("Thank god" yells the crowd), think about this beforehand.
You have time during your commute, over lunch or even in the shower.
When things start coming together use any other spare time to create
your "procedure". It might not be perfect but it will make the stress
managable and it might, with good logs, help you do what you need to
do.... Which is better than running around like a headless chicken
while the sky falls......
Original Tutorial
Submitted by nokia for TheTAZZone-TAZForum
Originally posted on March 4th, 2006 here
Do not use, republish, in whole or in part, without the consent of
the Author. TheTAZZone policy is that Authors retain the rights to the
work they submit and/or post...we do not sell, publish, transmit, or
have the right to give permission for such...TheTAZZone merely retains
the right to use, retain, and publish submitted work within it's
Network.