With it’s simplicity of use and the elegance it provides, Vagrant proves to be an interesting tool for a security professional. In this article I’ll give a basic overview so that you can start using it. So what is exactly Vagrant? Think of it as a wrapper around your virtualization provider like VMware or Virtual Box. By using Vagrant you get greater control and flexibility in creating, and managing multiple virtual machines at the same time.

To understand better let’s take an use case. Imagine that you are building an IDS lab. You would typically require 3 machines – Server, Sensor and Monitoring machine. Now to create this setup in Virtual Box you need to perform a series of repetitive tasks - installing the operating system, applications, define networking etc. Once the setup is done, you need to bring up and manage these machines individually. All these activities kind of take your focus away from the core of what you might have wanted– like testing some Snort rules or understanding how Bro IDS works. Wouldn’t it be great if these repetitive tasks can be automated, so that you can focus on stuff that matters?

This is where Vagrant comes in. With Vagrant you simply write a configuration file to define these 3 machines. You can subsequently bring them up, install the OS, provision them and manage them using a handful of “vagrant” commands. So with Vagrant, you move from managing individual machines to managing an environment. Here’s a simple visualization of how Vagrant works:

vagrant complete

#1 - The VagrantFile

The VagrantFile is an ASCII based text file where you specify the configuration of the virtual machine you require. If you need 10 machines for your lab then you need to specify them sequentially in the VagrantFile.
For every VM you can specify:

Physical properties of the machine (CPU, RAM etc)
Network Interfaces
Shared Folders (Folders shared between host and the VM)
Hostname/IP Address
Details of software to be installed on the VM – In Vagrant terms this is called provisioning.

Here is a snippet from the VagrantFile, to create a simple Ubuntu virtual machine. The syntax is based on Ruby. But even if you don’t know Ruby at all it is fairly straight to understand and implement.

#- This line indicates the beginning of configuration.  
Vagrant.configure(2) do |config| 
#- Begin configuration for my-ubuntu-machine 
config.vm.define "My Ubuntu Machine" do |my-ubuntu-machine| 
#- Specifying the “box” required for this machine and the URL to get it from.  
my-ubuntu-machine.vm.box="precise64"
my-ubuntu-machine.vm.box_url ="http://files.vagrantup.com/precise64.box"
#- Setting the hostname, IP and SSH port forwarding  
my-ubuntu-machine.vm.hostname="ubuntu-machine"
my-ubuntu-machine.vm.network "private_network", ip: "10.1.1.10"
my-ubuntu-machine.vm.network "forwarded_port", guest: 22, host: 2210  
#- Provisioning or software/config update – I am just running apt-get update to get the latest versions of packages  
my-ubuntu-machine.vm.provision "shell", inline: -SHELL 
#!/usr/bin/env bash 
apt-get update /dev/null 21 
SHELL 
end  
#- Add more VMs as required… 
#- Below line indicates end of configuration. 
end

#2 – The Vagrant Boxes

A box is file that represents a virtual machine. Think of it as an appliance that you can plug in to Vagrant and start running. Boxes are available for download from vagrantbox.es and the Hashicorp site. In addition to downloading boxes you can also create your own boxes and use them.

#3 – Virtualization Providers

This would typically be any software that provides an emulated environment for running different operating systems. Popular options are Vmware Player/Workstation/Fusion, and VirtualBox. Vagrant works free and fine with VirtualBox. However if you want to use it with VMware, you need to purchase additional licenses.

#4 – The Vagrant Software

The Vagrant software:

Reads the VagrantFile,
Downloads the boxes (if not already available)
Installs the required applications – For this it takes the help of
Creates VMs in the Virtualization provider
Provisions the machines as required - For simple software installations you can use shell scripts. Where shell scripts get complicated, Vagrant also provides support for Configuration management tools such as Chef, Puppet, Salt andAnsible.
Provides capabilities for managing the VMs using the “vagrant” command.

#5 – Plug-ins

Vagrant provides extensibility with the use of Plug-ins (not shown in above diagram). As a developer you might be interested to add new vagrant commands, provide easier options to configure a host, add support for provisioners etc. In fact, Vagrant itself ships with a set of core plug-ins that work behind the scenes to configure and bring up your VM. You can find the list of core plug-ins here. Plug-ins are developed using Ruby and are implemented as self-contained packages (Rubygems).

To know more about Vagrant refer the Vagrant documentation or the very readable book by Mitchell Hashimoto – Vagrant Up and Running.

The first step is to obtain the MFT. It can help answer the question – has any file been created on the file system between two times?

Obtain the MFT

The tool that I use for obtaining MFT is RawCopy.exe. It is a command line based tool, which interfaces with low level disk I/O. This allows the tool to even extract NTFS files that are locked by Windows.

Instructions	Screenshots
Download RawCopy.exe from GitHub
You will see the following files in the downloaded ZIP
Open Command prompt and run rawcopy.exe D:0 F:\output_folder The User Access Control popup will be shown. Click “Yes”. In this example, the $MFT file from D:\ partition will be copied under F:\output_folder.

Convert MFT –> BodyFile –> CSV

Instructions	Screenshots
Download and Install Python 2.7 The installation is simply point and click.
Add the path of python.exe to Environment variables This step is required so that you are able to run python from any directory.
Download analyzeMFT.py
You will see the following files after extracting the ZIP
Install analyzeMFT.py python setup.py install
Generate the Body File The body file is an intermediate file when creating a timeline of file activity. It is a pipe ("\|") delimited text file that contains one line for each file. The body file will have the following fields: MD5\|name\|inode\|mode_as_string\|UID\|GID\|size\|atime\|mtime\|ctime\|crtime To generate the body file run: analyzeMFT.py -f MFT1 -b MFT_body --bodystd –bodyfull This will take the $MFT (renamed to MFT1 for simplification), convert it to a body file and write the output to MFT_body.
Convert the BodyFile to CSV

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

#!/usr/bin/python 
############################################################################### 
# This script will parse a timeline file to create a Pcap representation 
# of the timeline for analysis in wireshark using the built-in features 
# such as profiles, colorization, filtering, and annotation. 
# 
# The basic process is to open the timeline file, parse each line, and 
# output a host file and Pcap that can be used together. Each entry in 
# the timeline is processed to create an individual tcp packet. 
# Characteristics of the packet are mapped to the timeline entry information. 
# For instance: 
# Source IP = Hostname 
# Destination IP = Target File 
# Source Port = Unused 
# Destination Port = Timeline Protocol Identifier 
# TCP Flags = MACB Flags 
# 
# This is just an example of the mapping power of using tcp. Other fields 
# of the packet that may be useful are IPID, sequence numbers, options field, and more. 
# 
# Limitations of this tool include: 
# 
# The host file is whitespace delimited so any whitespace characters are 
# interpreted as the end of the host name. This causes file names with white 
# space to be displayed improperly. This is a minor inconvenience as it is 
# remedied by importing the timeline parsing lua dissector which allows the 
# analyst to insert fields into the display to include the correct filename. 
# 
# The maximum payload of a tcp packet is 65,535 bytes. Timeline entries that 
# exceeded this maximum length were not expected but prefetch files have a 
# large amount of metadata included with them. This could be resolved by 
# creating tcp sessions for this information but that paricular solution 
# defeats the purpose of use which is easy display, colorization and marking 
# of timeline data. 
# 
############################################################################# 
# imports for required libraries 
import argparse 
import ntpath 
import logging 
logging.getLogger("Scapy").setLevel(1) 
# import the Scapy module for packet generation 
from scapy.all import *
#from scapy.utils import  
# import datetime and pytz to adjust for timezone offset 
from datetime import datetime, timedelta 
from pytz import timezone 
import pytz 
# Create a parser to read in command line arguments 
parser = argparse.ArgumentParser(description='Convert a timeline file into Pcap') 
  
# Add argument for input timeline file 
parser.add_argument('-f','--file', help='The timeline file to parse') 
# Add argument for time zone 
parser.add_argument('-z','--tzone', help='Timezone offset of the timeline in pytz format') 
# Add argument for output Pcap file 
parser.add_argument('-o','--outfile',help='Output filename for the resulting Pcap',default='./stimeline.Pcap') 
# Add argument for hostname 
parser.add_argument('-hn','--hostname',help='Name of the source host for this evidence',default='computer') 
# Add argument for hostfile 
parser.add_argument('-hf','--hostfile',help='Output filename for the resulting host file',default='./hosts') 
# Parse arguments for use 
args = parser.parse_args() 
# Open the host file with write access 
hostFile = open(args.hostfile, 'w') 
# Set the location of the Pcap file 
PcapFile = args.outfile 
# Set the port to represent timeline protocol traffic 
# this must match the port used in the target lua dissector 
timelinePort = 7143
# Create a timezone object for use in setting packet timestamp 
tz = timezone(args.tzone) 
# Set the starting network address for the packet capture 
# Currently, this packet capture will be able to process 56,634 
# unique files 
network = "172.16."
# Set the value of the third octet 
three = 0
# Set the value of the fourth octet, reserving .1 for the source host 
four = 2
  
# Set the source host information (IP and hostname) 
srcIP = "172.16.0.1"
srcHost = args.hostname 
# Create a packet list object to hold the crafted packets 
pkts = scapy.plist.PacketList() 
# Write the first entry in the host file, identifying the source of evidence 
hostFile.write(srcIP + "\t" + srcHost + "\n") 
#print(srcIP + "\t" + srcHost + "\n") 
# Dictionary to determine file/IP utilization 
files = dict() 
# Helper function for mapping months to month numbers 
def getMonthNumberFromShortMonth(shortMonth): 
    months={ "jan" : 1, "feb" : 2, "mar" : 3, "apr" : 4, "may" : 5, "jun" : 6, "jul" : 7, "aug" : 8, "sep" : 9, "oct" : 10, "nov" : 11, "dec" : 12 } 
    return months[shortMonth.lower()] 
  
#- rampa 
pktdump = PcapWriter(PcapFile,append=True, sync=True) 
  
# Open the timeline file with read permissions for parsing 
with open(args.file, 'r') as file: 
#{ 
    # For each line, parse the individual elements 
    for line in file: 
    #{ 
        elements = line.split(',') 
        date = elements[0] 
        size = elements[1] 
        macb = elements[2] 
        mode = elements[3] 
        uid = elements[4] 
        gid = elements[5] 
        meta = elements[6] 
        fname = elements[7] 
  
        # if the date element is "Date" then we are reading the header, skip 
        if (date.lower() == "date"): 
        #{ 
            continue
        #} 
  
        # Create a short filename by splitting at the last forward slash 
        short_fname_arr = fname.rsplit('/',1) 
  
        # Parse the macb flags and set tcp flags accordingly 
        # M=Push, A=Ack, C=Urg, B=Syn 
        P = "P" if macb[0].upper() == "M" else "" # M 
        A = "A" if macb[1].upper() == "A" else "" # A 
        U = "U" if macb[2].upper() == "C" else "" # C 
        S = "S" if macb[3].upper() == "B" else "" # B 
  
        # Assemble the resulting tcp flags into a single field 
        flags = S + A + P + U 
  
        # Check the length of the short name array, if < 2 then 
        # filename is first element, otherwise second 
        if (len(short_fname_arr) < 2): 
        #{ 
            short_fname = short_fname_arr[0][:-2] 
        #} 
        else: 
        #{ 
            short_fname = short_fname_arr[1][:-2] 
        #} 
  
        # Check to see if we already have an entry for this file, since 
        # multiple files may have the same name, we must use full name 
        if fname in files: 
        #{ 
            # If file already exists, reuse IP address 
            ip = files[fname] 
        #} 
        else: 
        #{ 
            # If not, then grab another IP, add it to the dictionary 
            if (four == 254): 
                if (three == 255): 
                    sys.exit('Timeline exceeded 56,654 entries...') 
                else: 
                    four = 0
                    three += 1
            else: 
                four += 1
  
            ip = network + str(three) + "." + str(four) 
            files[fname] = ip 
            # Add a new entry to the host file 
            hostFile.write(ip + "\t" + short_fname + "\n") 
            #print(ip + "\t" + short_fname + "\n") 
        #} 
  
        # Parse the date from the entry for use in Scapy 
        dateElements = date.split(' ') 
        timeElements = dateElements[4].split(':') 
        # If the timezone parameter is not null then use passed in parameter 
        # otherwise use None 
        if (timezone != None): 
            pktTime = datetime(int(dateElements[3]),getMonthNumberFromShortMonth(dateElements [1]),int(dateElements[2]),int(timeElements[0]),int(timeElements[1]),int (timeElements[2]),0,tzinfo=tz) 
        else: 
            pktTime = datetime(int(dateElements[3]),getMonthNumberFromShortMonth(dateElements [1]),int(dateElements[2]),int(timeElements[0]),int(timeElements[1]),int (timeElements[2]),0,None) 
  
        # Assemble packet payload constructing null terminated string from 
        # timeline file entry data 
        data = date + "\x00" + size + "\x00" + macb + "\x00" + mode + "\x00" + uid + "\x00" + gid + "\x00" + meta + "\x00" + short_fname + "\x00" + fname + "\x00" # Create an IP packet with TCP transport and data payload 
        p=IP(src=srcIP,dst=ip)/TCP(sport=timelinePort,dport=9999,flags=flags)/Raw(load=data) 
        # Set packet timestamp to constructed value 
        p.time = int(pktTime.strftime("%s")) 
  
        # Append packet to packet list 
        pkts.append(p) 
  
        # Write the Pcap file out to disk 
        pktdump.write(pkts) 
        #wrPcap(PcapFile,pkts) 
    #} 
#} 
# Close open resources 
hostFile.close() 
file.close()

My Journey into Information Security

Search This Blog

Tuesday 6 October 2015

5 things to know before you “vagrant up”