/usr/local/lp/apps/malre/scripts/ Current File: //usr/local/lp/apps/malre/scripts/issues.pl
#! /usr/bin/perl
### iterate through files that we have identified as having issues

use strict;
use warnings;

my $d_progdir = __FILE__;
$d_progdir = ( readlink($d_progdir) || $d_progdir );
my @progdir = split( m/\//, $d_progdir );
pop( @progdir );
$d_progdir = join( "/", @progdir );

require( $d_progdir . '/escape.pm' );
require( $d_progdir . '/stats.pm' );
require( $d_progdir . '/lock_file.pm' );
require( $d_progdir . '/json_io.pm' );

my $ref_issues;

=pod

### Purpose of this script:

The purpose of this script is to read, modify, and write the data object that will be used to store information about files - both those found by the various scanners that we will be using, and those manipulated by the user through wrapper scripts for commands such as "chmod" and "rm".

When a file is found by a scan, it will be referred to as a "problem"; when a file is seen as a result of a command the user ran, it will be referred to as an "action". We want to track both of these things - not just so that we can report them to the customer, but also because it will help us keep track of what problems are left to be addressed. When an action resolves a problem, it will be reffered to as a "remedy". The overarching goal here is to track what problems have been remedied, and what problems are still left to be addressed.

Among the user actions available will be the options to "ignore" a file and to "leave" a file. "Ignoring" a file will be the user's way of indicating that they do not believe that action needs to be taken on the file - for example, a false positive by a malware scanner. "Leaving" a file will be the user's way of indicating that they are unsure whether or not the file is actually malicious, and as a result it needs to be left for the customer to make that assessment.

### Processing scan results:

During the scanning process, scan results are captured by the file ./scripts/stats.pm (either as part of the process from ./scripts/find.pl or ./scripts/get_stats.pl, depending on the scan) and temporarily stored in a format where the information regarding the files that were found is separated by null characters. This format was chosen because it's easy to append to - instead of bogging down the scanning process with manipulating a large data object, just append to the end of a file and call it good.

At the end of the scanning process, this data is read and converted into an intermediary JSON style object. At this point, the JSON data from the object that we're using to store details about files and the problems and actions that relate to them is read from disk and then the data from the intermediary object is entered into the final JSON object. Once that is complete, it will be written back to disk with the new data in place.

The first step in this assimilation is to check the full path of the file to see if it already exists in the object by cross referencing the information in $ref_issues->[0] (see below for further details). From that point forward in the process, the file will be referred to by a reference number. This reference number is used in $ref_issues->[1] to store important data on the file including its file name and a snapshot of its stats at any of the ctimes at which we've seen it.

As we assess the data for each file we will look at which scan brought it to our attention and whether or not the user has already taken actions that might remedy the problem described by that scan. To explain further: an unfortunate quirk of this process is that scan results will be assimilated at the end of the scanning process (meaning that we cannot necessarily collect file stats until after the scan has finished), but it's possible for the user to see those results before they complete and thus act on them before the data from them is stored. If, for example, a user chmods a malicious file to 000 before the end of the scan, the wrapper scripts will allow us to capture the ctime of the file both before and after their action. If the ctime of the file captured after the scan is equal to or later than the user's action, and the permissions are the same as from the user's action, we can safely assess that that is one scan result that has already been remedied, and thus further action does not need to be taken on it.

Whether or not a problem is present for a file will not only be stored in $ref_issues->[1]->[$num]->[3]->{$ctime}->[8], but will also be stored within $ref_issues->[2]->[$prob]->{$num}->{$ctime} - this will allow us to easily create lists of files associated with each type of problem that have yet to be remedied, thus prompting the user on what they can do next.

Finally, if we are not sure that the ctime captured for the file was not associated with an action taken by the user, we will store it within $ref_issues->[4] so that we can reference it later when checking logs.

### Processing user actions

The wrapper scripts for user actions will rely on the backup functionality of Stat Watch. Where possible, a backup of the impacted file will be taken both before and after the action the user desired. (This becomes complicated if an action is being applied recursively to an entire directory, but we'll do our best).

Similarly to how scan results are processed, the data for the file will be converted into an intermediary format and then assimilated into the final JSON object. For each action taken against a file, an assessment will be made regarding whether that action remedies any of the problems that are currently associated with a file. For example, If a file's permissions are changed from 777 to 644, that remedies the fact that the file was world writable, but does not remedy the fact that the file was detected by maldet to have potentially malicious content.

The assimilation will contain two sets of data for the file - one from before the user's change took place and one from after. As with scan results, if we are not sure that the ctime captured for the file previous to the user's action was not associated with another action taken by the user, we will store it within $ref_issues->[4] so that we can reference it later when checking logs.

=cut

=pod

### Notes on the structure of the JSON object:

### Top level reference points:

$ref_issues->[0] <- a series of hashes emulating the directory tree. This is what we'll use to look up a file
$ref_issues->[1] <- an array containing details on the files. each file will have a single item in the array
$ref_issues->[2] <- an array of problem types, each containing a hash of numbers associated to files
$ref_issues->[3] <- an array of action types, each containing a hash of numbers associated to files
$ref_issues->[4] <- an hash of ctimes that were not associated with user actions

### Looking up a file by name

$ref_issues->[0]-> a copy of the directory tree making it easy to look up files
$ref_issues->[0]->{DIRECTORY TREE PATH}->{$v_name}->{'.'} <- the number that we've associated with the file

### Details about the files themselves

$ref_issues->[1]->[$num]->[0] <- the name of the file (with full path)
$ref_issues->[1]->[$num]->[1] <- The latest ctime we've seen for the file, or, if the file didn't exist, the earliest stamp for when we tried to get stats
$ref_issues->[1]->[$num]->[2] <- The latest ctime for when the file was present
$ref_issues->[1]->[$num]->[3] <- a hash of ctimes at which we've seen the file
$ref_issues->[1]->[$num]->[3]->{$ctime}->[0] <- the permissions of the file in 4 digit format
$ref_issues->[1]->[$num]->[3]->{$ctime}->[1] <- The user number of the file
$ref_issues->[1]->[$num]->[3]->{$ctime}->[2] <- the group number of the file
$ref_issues->[1]->[$num]->[3]->{$ctime}->[3] <- the size of the file in bytes
$ref_issues->[1]->[$num]->[3]->{$ctime}->[4] <- Most recently seen mtime
$ref_issues->[1]->[$num]->[3]->{$ctime}->[5] <- Most tecent time at which stats were collected
$ref_issues->[1]->[$num]->[3]->{$ctime}->[6] <- Whether it's a file or a directory
$ref_issues->[1]->[$num]->[3]->{$ctime}->[7] <- Wether the agent made the change (1), or something unknown made the change (0)
$ref_issues->[1]->[$num]->[3]->{$ctime}->[8] <- an array of the different problem lists, where 0 indicates they're not on the list
$ref_issues->[1]->[$num]->[3]->{$ctime}->[9] <- An array of actions that the user might have taken on the file

### Problem lists

$ref_issues->[2]->[$prob] <- Each number will correlate with a type of problem (see below for details)

Each of these will include a hash of file reference numbers.
These will then contain an array with the ctime, and then a 1 or a 0 indicating whether or not the problem is still present
The script will only modify it if the current ctime is higher than the one that's present; it will discard the old ctime and replace it with the current ctime
Example:

$ref_issues->[2]->[$prob]->{$num}->[0] <- The ctime of the most recent time the file was assessed
$ref_issues->[2]->[$prob]->{$num}->[1] <- A "1" or a "0" indicating whether or not the problem has been remedied

Note, user actions have a similar structure within $ref_issues->[2]

### The array under $ref_issues->[1]->{$num}->[3]->{$ctime}->[8] (problem lists)
These numbers also correlate to $ref_issues->[2]
Problems are only identified when incorporating scans

0  <- Checkers scan - "0" if it was not found, $result if it was
	Remedied by: 0,1,3,4,5,6,7
1  <- Checkers WordPress md5 - "0" if it was not found, "1" if it was
	Remedied by: 0,1,3,4,5,6,7
2  <- Checkers Wordpress Extra - "0" if it was not found, "1" if it was
	Remedied by: 0,1,3,4,6,7
3  <- Maldet Uncertain - "0" if it was not found, $result if it was
	Remedied by: 0,1,3,4,5,6,7
4  <- Maldet - "0" if it was not found, $result if it was
	Remedied by: 0,1,3,4,5,6,7
5  <- World Writable - "0" if it was not found, "1" if it was
	Remedied by: 2 (only if it is no longer world writable afterward)
6  <- Files with unusual characters in their name - "0" if it was not found, "1" if it was
	Remedied by: 0,1,6,7
7  <- File that is a symbolic link - "0" if it was not found, "1" if it was
	Remedied by: 0,6,7
8  <- Directory containing multiple symlinks - "0" if it was not found, $number_of_symlinks if it was
	Remedied by: 0,1,6,7
9  <- Stat Watch new file
	Remedied by: 0,1,6,7
10 <- Stat Watch removed file
	Remedied by: 4,5,6,7
11 <- Stat Watch modified file
	Remedied by: 0,1,3,4,5,6,7
12 <- Stat watch modified permissions file
	Remedied by: 2,5,6,7

### Action lists

The array under $ref_issues->[1]->{$num}->[3]->{$ctime}->[9] (action lists)
Numbers correlate to $ref_issues->[3]
Actions are only identified when assessing user actions

0 <- Removed
	Doesn't remedy: 
	Unremedied by: Being present again
1 <- Disabled (given 000 permissions)
	Doesn't remedy: 
	Unremedied by: Being seen with any permissions other than 000
2 <- Permissions Modified (but not disabled)
	Doesn't remedy: Can only remedy world writable
	Unremedied by: 
3 <- Modified (any interaction from the commands "vim", or "nano" that might have changed the contents of the file)
	Doesn't remedy: 5,6,7,8,12
	Unremedied by: Being found by a scan again
4 <- Replaced (any interaction from the commands "cp", or "mv" that might have changed the contents of the file)
	Doesn't remedy: 5,6,7,8,12
	Unremedied by: Being found by a scan again
5 <- Restored (From backups taken by Stat Watch / Malre )
	Doesn't remedy: 5,6
		Finds details on what problems were present at the time stamp where they were at
	Unremedied by: Being found by a scan again
6 <- Ignored (we assessed that there was nothing wrong with it)
	Doesn't remedy: 
	Unremedied by: Being found by a scan again with a new value
7 <- Customer assessment (We are leaving it up to the customer to make an assessment)
	Doesn't remedy: 
	Unremedied by: Being found by a scan again with a new value

### Note: file references within $ref_issues->[1]->[$num] are not created by default for directories unless those directories are specifically being mentioned

=cut

sub fn_get_num {
### Given the full path to a file, return the number associated with it
### Given a file that does not already have a number, create a number
	my $v_file = $_[0];
	my $num = scalar @{ $ref_issues->[1] };
	my @v_file = split( m/\//, $v_file );
	my $path = $ref_issues->[0];
	while ( defined $v_file[0] ) {
		my $dir = shift(@v_file);
		if ( ! exists $path->{$dir} ) {
			$path->{$dir} = {};
		}
		$path = $path->{$dir};
	}
	if ( exists $path->{'.'} ) {
		$num = $path->{'.'};
	} else {
		$path->{'.'} = $num;
		push( @{ $ref_issues->[1] }, $v_file );
		$ref_issues->[1]->[$num]->[0] = $v_file;
	}
	return $num;
}

sub fn_import_file {
### Given a file name and a reference object for the file, incorporate that data into the json object
### This function will only cover the portions that are mutually present when assimilating either scan results on user actions
### return the $num and $ctime for the file (so that the rest of the data can be appropriately incorporated)
	my $v_file = $_[0];
	my $ref_file = $_[1];

	my $num = fn_get_num($v_file);
	my $ctime = $ref_file->{'ctime'};

	### Typically, we're storing details by the ctime of the file, but if the file doesn't exist, this becomes more complicated, as instead of the ctime, we want to store by the first timestamp that we're aware of where it did not exist
	if ( defined $ref_issues->[1]->[$num]->[1] && $ref_file->{'ftype'} eq '' ) {
	### If we've seen the file before, and this time when we tried to get stats, the file did not exist
		if ( ! defined $ref_issues->[1]->[$num]->[2] || $ref_issues->[1]->[$num]->[1] > $ref_issues->[1]->[$num]->[2] ) {
		### If the file also did not exist the last time we saw it
			if ( $ctime < $ref_issues->[1]->[$num]->[1] && ( ! defined $ref_issues->[1]->[$num]->[2] || $ctime > $ref_issues->[1]->[$num]->[2] ) ) {
			### If the stamp of us trying to get stats is earlier than the previous stamp we saw, but later than the ctime from the last time the file was actually present
			### Change the latest timestamp to the earlier timestamp
				my $v_latest = $ref_issues->[1]->[$num]->[1];
				$ref_issues->[1]->[$num]->[1] = $ctime;
				$ref_issues->[1]->[$num]->[3]->{$ctime} = $ref_issues->[1]->[$num]->[3]->{$latest};
				delete $ref_issues->[1]->[$num]->[3]->{$latest};
			} elsif ( $ctime > $ref_issues->[1]->[$num]->[1] ) {
			### if the stamp of us trying to get stats is later than the the latest stamp we saw, change the ctime to match the earlier stamp
				$ctime = $ref_issues->[1]->[$num]->[1];
			} elsif ( $ctime < $ref_issues->[1]->[$num]->[2] ) {
			### If the stamp of us trying to get stats is earlier than the last time the stamp existed
				### Optimally we would have code that continually seeks through trying to find the largest ranges of where the file did not exist
				### but because us recieving earlier timestamps is an edge case already, I don't see any reason to flesh this out.
				### We will just keep the ctime as it is and if there end up duplicate entries, meh.
			}
		} else {
		### If the last time we saw the file, it DID exist
			### Once again, edge cases upon edge cases here
		}
	}

	### Update the "latest" and "exists" stamps
	if ( ! defined $ref_issues->[1]->[$num]->[1] || $ctime > $ref_issues->[1]->[$num]->[1] ) {
		$ref_issues->[1]->[$num]->[1] = $ctime;
		if ( $ref_file->{'ftype'} ne '' ) {
			$ref_issues->[1]->[$num]->[2] = $ctime;
		}
	}

	### Update the file details
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[0] = $ref_file->{'perms'};
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[1] = $ref_file->{'user'};
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[2] = $ref_file->{'group'};
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[3] = $ref_file->{'size'};
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[4] = $ref_file->{'mtime'};
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[5] = $ref_file->{'stamp'};
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[6] = $ref_file->{'ftype'};
	if ( ! exists $ref_issues->[1]->[$num]->[3]->{$ctime}->[8] ) {
		@{ $ref_issues->[1]->[$num]->[3]->{$ctime}->[8] } = ( 0,0,0,0,0, 0,0,0,0,0, 0,0,0 );
	}
	if ( ! exists $ref_issues->[1]->[$num]->[3]->{$ctime}->[9] ) {
		@{ $ref_issues->[1]->[$num]->[3]->{$ctime}->[9] } = ( 0,0,0,0,0, 0,0,0 );
	}

	return( $num, $ctime );
}

sub fn_initialize_json {
### If the json structure has yet to be initialized, do so now
	if ( ! defined $ref_issues->[1] ) {
		@$ref_issues->[1] = [];
	}
	if ( ! defined $ref_issues->[0] ) {
		$ref_issues->[0] = {};
	}
	if ( ! defined $ref_issues->[2] ) {
		### Create the section for problems
		$ref_issues->[2]->[0]  = {};
		$ref_issues->[2]->[1]  = {};
		$ref_issues->[2]->[2]  = {};
		$ref_issues->[2]->[3]  = {};
		$ref_issues->[2]->[4]  = {};
		$ref_issues->[2]->[5]  = {};
		$ref_issues->[2]->[6]  = {};
		$ref_issues->[2]->[7]  = {};
		$ref_issues->[2]->[8]  = {};
		$ref_issues->[2]->[9]  = {};
		$ref_issues->[2]->[10] = {};
		$ref_issues->[2]->[11] = {};
		$ref_issues->[2]->[12] = {};
	}
	if ( ! defined $ref_issues->[3] ) {
		### Create the section for actions
		$ref_issues->[3]->[0] = {};
		$ref_issues->[3]->[1] = {};
		$ref_issues->[3]->[2] = {};
		$ref_issues->[3]->[3] = {};
		$ref_issues->[3]->[4] = {};
		$ref_issues->[3]->[5] = {};
		$ref_issues->[3]->[6] = {};
		$ref_issues->[3]->[7] = {};
	}
	if ( ! defined $ref_issues->[4] ) {
		$ref_issues->[4] = {};
	}
}

sub fn_get_latest {
### Given an entry within the problem or action branches and a file number, return the latest timestamp and the value of that stamp
### $_[0] is "2" or "3", for problems or actions
### $_[1] is the item number from the problem or acton list
### $_[2] is the file number
	my $branch = $_[0];
	my $prob = $_[1]; ### While "prob" is used here, this could just as easily be a number from the actions array as it is a number from the problems array
	my $num = $_[2];
	my $latest = 0;
	my $value = 0;
	if ( exists $ref_issues->[$branch]->[$prob]->{$num} && defined $ref_issues->[$branch]->[$prob]->{$num} ) {
		$latest = $ref_issues->[$branch]->[$prob]->{$num}->[0];
		$value = $ref_issues->[1]->[$num]->[3]->{$latest}->[$branch + 6]->[$prob];
	} elsif ( exists $ref_issues->[$branch]->[$prob]->{$num} ) {
		### If it's not defined, get rid of it
		delete $ref_issues->[$branch]->[$prob]->{$num};
	}
	return( $latest, $value );
}

sub fn_assess_problem {
### Given a file number, a problem number, a ctime, and a "1" or a "0" (whether or not the problem is present), assess if we can make an update and then do so if we can
### Optional fifth argument: a description (such as maldet results)
	my $num = $_[0];
	my $prob = $_[1];
	my $ctime = $_[2];
	my $status = $_[3];
	my $descrip = ( $_[4] || $status );
	my $b_change = 0;
	( my $latest, my $value ) = fn_get_latest( 2, $prob, $num );
	if ($latest) {
		my $b_cont = 1;
		( my $ign_latest, my $ign_value ) = fn_get_latest( 3, 5, $num );
		if ($ign_latest) {
			( my $rem_latest, my $rem_value ) = fn_get_latest( 3, 0, $num );
		}
	}









	##### Anything after this point in this function is likely outdated
	if ( exists $ref_issues->[2]->[$prob]->{$num} ) {
		### If the file has been previously ignored, we only un-ignore it for two reasons:
			### If it was detected on a scan where we've never detected it previously
			### If it was previously disabled or removed, but now it's back
		my $b_cont = 1;
		if ( exists $ref_issues->[3]->[5]->{$num} ) {
			$b_cont = 0;
			if ( exists $ref_issues->[3]->[0]->{$num} && $ref_issues->[3]->[0]->{$num} < $ctime ) {
			### If it was removed in the past
				if ( $ref_issues->[1]->[$num]->[3]->{$ctime}->[6] ne '' ) {
				### But it's present now
					delete $ref_issues->[3]->[0]->{$num}
				}
			}
		}
		$latest = (keys( %{ $ref_issues->[2]->[$prob]->{$num} } ))[0];
		if ( $ctime > $latest ) {
		### Only greater than, not equal to. Equal to would result in us re-declaring ignored things as problems
			delete $ref_issues->[2]->[$prob]->{$num}->{$latest};
			$ref_issues->[2]->[$prob]->{$num}->{$ctime} = $status;
			$b_change = 1;
		}
	} else {
		$ref_issues->[2]->[$prob]->{$num}->{$ctime} = $status;
		$b_change = 1;
	}
	$ref_issues->[1]->[$num]->[3]->{$ctime}->[8]->[$prob] = $descrip;
	return $b_change;
}

sub fn_pull_scan {
### Given the null character separated file generated by stats.pm and the file storing the json object
### Incorporate the data from the one into the other
	my $f_report = $_[0];
	my $f_json = $_[1];
	my $ref_report = fn_report_to_json($f_report);
	$ref_issues = fn_read_json($f_json);
	fn_initialize_json();
	for my $ref_file ( @{ $ref_report } ) {
		### Find a number to associate with the file if the file does not already exist
		my $v_file = $ref_file->{'path'};
		( my $num, my $ctime ) = fn_import_file($v_file);
		if ( ! exists $ref_issues->[1]->[$num]->[3]->{$ctime}->[7] ) {
			### Only if there is not already a value already in place for who did it should we note that we do not know
			$ref_issues->[1]->[$num]->[3]->{$ctime}->[7] = 0;
		}

		### Depending on the type and details, we need to record different data
		##### I'm not one hundred percent sure on anything that's below this line
		if ( $ref_file->{'type'} == 'chars' ) {
			if ( $ref_file->{'ftype'} ne '' && substr( $ref_file->{'perms'}, 1 ) ne '000' ) {
				### The file has strange characters and is still an issue
				fn_assess_problem( $num, 6, $ctime, 1 );
			} else {
				### The file was either removed or has 000 permissions. not an issue.
				fn_assess_problem( $num, 6, $ctime, 0 );
			}
		} elsif ( $ref_file->{'type'} == 'wx' ) {
			if ( $ref_file->{'ftype'} ne '' ) {
				my $world = substr( $ref_file->{'perms'}, 3 );
				if ( $world =~ /^[2367]$/ ) {
					### Still a problem - lets see if we should note it as an issue
					fn_assess_problem( $num, 5, $ctime, 1 );
				} else {
					fn_assess_problem( $num, 5, $ctime, 0 );
				}
			} else {
				fn_assess_problem( $num, 5, $ctime, 0 );
			}
		} elsif ( $ref_file->{'type'} == 'symlink' ) {
			if ( $ref_file->{'ftype'} ne '' ) {
			}
		}




		##### everything within this function beyond this point might be outdated or subject to needing change
		if ( $ref_file->{'type'} == 'symlink' ) {
			$ref_issues->[1]->{$num}->{'symlink'} = 1;
			$ref_issues->{'sym'}->{$num} = 1;
		}
		if ( $ref_file->{'type'} == 'symlink_dir' ) {
			$ref_issues->[1]->{$num}->{'symlink_dir'} = $ref_report->{$_file}->{'symlink_dir'};
			$ref_issues->{'sym_dir'}->{$num} = 1;
		}
		if ( $ref_file->{'type'} == 'maldet' ) {
			$ref_issues->[1]->{$num}->{'maldet'} = $ref_report->{$_file}->{'maldet'};
			if ( $ref_report->{$_file}->{'maldet'} =~ m/((inject|generic)\.|UNOFFICIAL)/ ) {
				$ref_issues->{'maldet_un'}->{$num} = 1;
			} else {
				$ref_issues->{'maldet'}->{$num} = 1;
			}
		}
		if ( $ref_file->{'type'} == 'checkers_scan' ) {
			$ref_issues->[1]->{$num}->{'checkers_scan'} = $ref_report->{$_file}->{'checkers_scan'};
			$ref_issues->{'checkers'}->{$num} = 1;
		}
		if ( $ref_file->{'type'} == 'checkers_wp' ) {
			$ref_issues->[1]->{$num}->{'checkers_wp'} = $ref_report->{$_file}->{'checkers_wp'};
			if ( $ref_report->{$_file}->{'checkers_wp'} =~ m/:md5/ ) {
				$ref_issues->{'checkers_wp_md5'}->{$num} = 1;
			} else {
				$ref_issues->{'checkers_wp_extra'}->{$num} = 1;
			}
		}


	}
}




sub fn_report_to_json_1 {
##### This is old and needs to be removed in its entirety once I'm sure that I no longer need it for reference purposes
### Given a report file from a malware scan, integrate the data from it into the json issues file
	my $f_report = $_[0];
	my $f_json = $_[1];
	my $ref_report = fn_report_to_json($f_report);
	$ref_issues = fn_read_json($f_json);
	for my $_file ( keys( %{ $ref_report } ) ) {
		my $num;
		if ( exists $ref_issues->{'toc'}->{$_file} ) {
			$num = $ref_issues->{'toc'}->{$_file};
		} else {
			$num = fn_get_num();
			$ref_issues->{'toc'}->{$_file} = $num;
		}
		$ref_issues->[1]->{$num}->{'name'} = $_file;

		### Get the current timestamps. If they're more recent than the timestamps from the report, don't reset anything
		my @v_stats;
		if ( -e $_file ) {
			if ( -l $_file ) {
		      	@v_stats = (lstat( $v_file ))[2,4,5,7,9,10];
			} else {
		      	@v_stats = (stat( $v_file ))[2,4,5,7,9,10];
			}
			push( @v_stats, time() );
			$v_stats[0] = sprintf "%04o", $v_stats[0] & 07777;
		}
		my $mtime_cur = 0;
		my $ctime_cur = 0;
		my $stamp_cur = 0;
		my $mtime_scan = $ref_report->{$_file}->{'mtime'};
		my $ctime_scan = $ref_report->{$_file}->{'ctime'};
		my $stamp_scan = $ref_report->{$_file}->{'stamp'};
		if ( ! @v_stats ) {
			if ( ! exists $ref_issues->{'removed'}->{$num} ) {
				$ref_issues->{'removed'}->{$num} = 1;
			}
			$ref_issues->[1]->{$num}->{'perms'} = $ref_report->{$_file}->{'perms'};
			$ref_issues->[1]->{$num}->{'user'} = $ref_report->{$_file}->{'user'};
			$ref_issues->[1]->{$num}->{'group'} = $ref_report->{$_file}->{'group'};
			$ref_issues->[1]->{$num}->{'size'} = $ref_report->{$_file}->{'size'};
			$ref_issues->[1]->{$num}->{'mtime'} = $ref_report->{$_file}->{'mtime'};
			$ref_issues->[1]->{$num}->{'ctime'} = $ref_report->{$_file}->{'ctime'};
			$ref_issues->[1]->{$num}->{'stamp'} = $ref_report->{$_file}->{'stamp'};
			##### If it's maldet, checkers, or char, add to the list of timestamps that we need to evaluate
		} else {
			if ( exists $ref_issues->{'removed'}->{$num} ) {
				### If it exists, it's no longer removed
				delete $ref_issues->{'removed'}->{$num};
			}
			##### Here we have the potential of three sets of stats - those from the last time malre saw the file, those from the scan, and those currently present
			##### We definitely want to make sure that all fo them are captured, but that gets taken care of later
			##### There are six patterns that these might be in:
				##### last = scan = present
					##### Trust the scan
				##### last = scan -> present
					##### something outside of us changed this since the scan
				##### last -> scan = present
					##### Trust the scan
				##### last -> scan -> present
					##### Something outside of us changed this since the scan
				##### scan -> last = present
					##### We changed this last
				##### scan -> last -> present
					##### something outside of us changed this since we last changed it
			my $mtime_doc = 0;
			my $ctime_doc = 0;
			my $stamp_doc = 0;
			if ( exists $ref_issues->[1]->{$num}->{'stamp'} ) {
				$mtime_doc = $ref_issues->[1]->{$num}->{'mtime'};
				$ctime_doc = $ref_issues->[1]->{$num}->{'ctime'};
				$stamp_doc = $ref_issues->[1]->{$num}->{'stamp'};
			}
			$mtime_cur = $v_stats[4];
			$ctime_cur = $v_stats[5];
			$stamp_cur = $v_stats[6];


			### If the file fits into one of the categories we're scanning for, document it appropriately.
			if ( exists $ref_report->{$_file}->{'000'} ) {
				### If it hasn't been disabled previously...
				if ( ! exists $ref_issues->{'disabled'}->{$num} ) {
					$ref_issues->[1]->{$num}->{'000'} = 1;
					$ref_issues->{'000'}->{$num} = 1;
				}
			}
			if ( exists $ref_report->{$_file}->{'chars'} ) {
				$ref_issues->[1]->{$num}->{'chars'} = 1;
				$ref_issues->{'chars'}->{$num} = 1;
			}
			if ( exists $ref_report->{$_file}->{'wx'} ) {
				$ref_issues->[1]->{$num}->{'wx'} = 1;
				$ref_issues->{'write'}->{$num} = 1;
			}
			if ( exists $ref_report->{$_file}->{'symlink'} ) {
				$ref_issues->[1]->{$num}->{'symlink'} = 1;
				$ref_issues->{'sym'}->{$num} = 1;
			}
			if ( exists $ref_report->{$_file}->{'symlink_dir'} ) {
				$ref_issues->[1]->{$num}->{'symlink_dir'} = $ref_report->{$_file}->{'symlink_dir'};
				$ref_issues->{'sym_dir'}->{$num} = 1;
			}
			if ( exists $ref_report->{$_file}->{'maldet'} ) {
				$ref_issues->[1]->{$num}->{'maldet'} = $ref_report->{$_file}->{'maldet'};
				if ( $ref_report->{$_file}->{'maldet'} =~ m/((inject|generic)\.|UNOFFICIAL)/ ) {
					$ref_issues->{'maldet_un'}->{$num} = 1;
				} else {
					$ref_issues->{'maldet'}->{$num} = 1;
				}
			}
			if ( exists $ref_report->{$_file}->{'checkers_scan'} ) {
				$ref_issues->[1]->{$num}->{'checkers_scan'} = $ref_report->{$_file}->{'checkers_scan'};
				$ref_issues->{'checkers'}->{$num} = 1;
			}
			if ( exists $ref_report->{$_file}->{'checkers_wp'} ) {
				$ref_issues->[1]->{$num}->{'checkers_wp'} = $ref_report->{$_file}->{'checkers_wp'};
				if ( $ref_report->{$_file}->{'checkers_wp'} =~ m/:md5/ ) {
					$ref_issues->{'checkers_wp_md5'}->{$num} = 1;
				} else {
					$ref_issues->{'checkers_wp_extra'}->{$num} = 1;
				}
			}

			### If we've never seen this file before, or we have seen it before but the ctime was more recent during the scan
			if ( $ctime_scan > $ctime_doc ) {
				### If the file is marked as disabled, but the current permissions are not 000, remove that status
				### If the file is marked as modified, remove that status
				### If the file is marked as permissions being modified, but the permissions problems detected during the scan still exist
			}

		}

		### Get the size and the timestamps in place
		$ref_issues->[1]->{$num}->{'size'} = $ref_report->{$_file}->{'size'};
		if ( ! exists $ref_issues->[1]->{$num}->{'stamps'}->{ $ref_report->{$_file}->{'mtime'} } ) {
			$ref_issues->[1]->{$num}->{'stamps'}->{ $ref_report->{$_file}->{'mtime'} } = 1;
		}
		if ( ! exists $ref_issues->[1]->{$num}->{'stamps'}->{ $ref_report->{$_file}->{'ctime'} } ) {
			$ref_issues->[1]->{$num}->{'stamps'}->{ $ref_report->{$_file}->{'ctime'} } = 1;
		}
		##### Don't do this is the current timestamps are more recent thant he ones from the report
		$ref_issues->[1]->{$num}->{'mtime'} = $ref_report->{$_file}->{'mtime'};
		$ref_issues->[1]->{$num}->{'ctime'} = $ref_report->{$_file}->{'ctime'};
		##### If the current timestamps aren't in the list of timestamps, add to the list of timestamps.

	}
	fn_write_json($ref_issues);
}

if ( $ARGV[0] && $ARGV[0] eq "--integrate-scan" ) {
	if ( $ARGV[1] && $ARGV[2] ) {
		fn_pull_scan( $ARGV[1], $ARGV[2] );
	} else {
		print "Need a file to pull from and a json file to integrate data to\n"
	}
}