Reminiscences of a Software Developer : 15 years, 15 mini-acheivements

I recently completed fifteen years a as software developer, and found it fit to list down and talk about some of the challenging problems I was able to solve over the years. This is a blog to list those down.
These are not the cutting edge of any tech, but small accomplishments that I personally feel very satisfied about. These solutions are not celebrated acheivements, solving them did not send stock prices soaring. They are part of everyday work of an engineer. These problems looked really hard to solve when I started on them and hence solving them instilled a sense of accomplishment, a smile back home, a story worth the record.
I am thankful for the opportunity to work on these problems and though am a little proud of having solved them, in no way I could claim novelty of the ideas, and would like to avoid any impression of sounding pompous. A better solution exists for all of them. Another engineer could have probably done a better job at solving them and could have solved them faster.
If you are that ‘another’ engineer, please get in touch because we are hiring.

1. Validating an applet using JNI from within the applet.
In 2015, a certain security software needed to validate certain things about a java applet. The mechanism was to use a Java TI agent (which is a debugging technique used by Java profilers) to get a DLL to run as a part of java runtime that ran that applet.
We needed to check if an applet based out of an unsigned jar is not setting Java’s security manager to NULL, so that it coud be allowed to access files in the operating system and do stuff a browser could otherwise do.
The challenge was to access the information about the applet from within that TI environment which was based on JNI. The code is very easy to write in Java, but we were in C++ and if you are still reading this you probably would know that one line of Java cde could translate to 10 lines of JNI code.
The challenge was to get the validation code right in C/C++. Took some research to understand how each line of Java code could be written in JNI. But the validation was fruitful at the end. We could catch the bad applets and allowed the good applets to do whatever they wanted.

2. Intercpeting an emil as soon as send button is pressed.
In 2009, we developed an MS Outlook based solution to detect leakage of sensitive content via emails. While software running at the level of network firewall could also detect leakage of sensitive data, it was not effective in case of encrypted traffic or people working from home.
Of course, an MS office plugin could be written to sniff contents of outgoing email, but the challenge was to do that in C/C++ (COM specifically) and not burden the user to deploy the office plugin but to install it ourselves. We hacked the registry and clapped in excitement when the plugin DLL showed up in list of Outlooks’ DLLs in process explorer. The next challenge was to intercept the outgoing email and registered an IMAPI sink to get notification about the email as soon as the send button was pressed. I still remembered the feeling of thrill when it dumped entire text of the email between HTML markups and ineterecpt was complete! Magic!

3. Correlate events from a minifilter driver to produce actionable security alerts
In 2007 we worked for a cybersecurity client who was looking to implement File Integrity Monitoring using a minifilter driver. We were given the minifilter and the user mode piece around it and needed to detect if monitored files were edited.
There were tens or even hundreds of events for one single ‘file edit’. The challenge was to just raise one event per user action resulting in a file edit. If notepad opened the file twice and edited the file twice, we wanted two events. If the user pressed ‘cltr-s’ a hundredth times while file remained open in notepad, we wanted one event.
There was no correlation between the create-write-close IRPs and user action. A single ctrl-s could trigger many such create-write-setInformation-close tuples. I was just one year out of college. We had two weeks time before the release to QA.
Eventually was able to pull it and there was a formal award for that work.
(I still wonder what the best way is to raise just one event per user action pertaining to a file edit)

4. Using time as a variable.
This was in 2010 when our DLP product needed to prevent a file from getting leaked by intercepting a save action to a remote drive. There were cases when the file could only be classified for sensitive content only after it was saved to the destination. So, we needed to undo the action of saving sensitive content to a remote drive. We were operating in a web of API hooks which were all over the place.
Some applications would simply delete the file they were writing to and write the new one afresh. We wanted to back-up a deleted file with the purpose of restoring it later should we find the new content sensitive. Throw away the backup if everything was good.
The challenge was we did not know if the user ws deleting the file deliberately, in which case we did not want a backup. We just could not keep backing up files when user was trying to free up space in a 512 MB thumb drive, for one in a millionth chance of detecting and having to prevent a DLP violation.
So we tracked the timestamp of various APIs that were called. We then profiled user actions vis-a-vis application triggered actions based on the time gap between those APIs. The assumption was that programs were fast and users were slow. It it was a programmatic operation, there would not be more than a few milliseconds of gap between return from one system call and entry into the next one. A user action, on another hand would show a gap of at least one second between API calls. If a delete was followed by a create within a few milliseonds, from the same thread, this was probably a replace attempt by the application and not a user deleting a file and creating a new one. We would take the backup in those cases. We could only take a backup when the Delete API was called so we had to clear the backup if the same thread did not come back with a Create API call within a few seconds. How could one check for a condition a few millseconds in the future? Even std::future was a thing of the future.
Very few of my old colleagues know about this peice. There was no review system back then, there was no design proposal, it went in as a bug fix and the code just ‘worked’.

5. Detecting what is printing.
This was around 2012 and I needed to fix a bug related to printing of sensitive content. Problem here was that the APIs we hooked to grab the text being printed and hence classify it, were not the ones which could be used to stop printing of the same sensitive content without the application going haywire and throwing dialogue boxes and in some cases crashing without saving.
The APIs which could be used to elegantly intercept a print job would just give us glyph buffers which coul not be classified by text-based rgular expressions . So, I was able to come up with an algorithm that used the font and character map information to reconvert the glypgh back to text. There is a patent for that ‘bug-fix’.

6. Separating .NET runtime code from shell code
In 2015, one of the features of a security product I worked on was to detect shell code getting executed from within an application. It would look at the address of previous instruction that executed from an API hook and try to determine if that came from a valid module loaded within the binary’s text section or from heap memory regions marked with RWX. In the second case, suspicion would increase that the hooked API was being called from shell code.
Except that a lot of valid software would allocate memory regions with RWX and execute code from them. Visual basic runtime, java runtime and .NET were examples of such technologies.
.NET was interesting, because one could actually use a windbg extensions and peek into a valid stack of .NET classes when given an address of an RWX heap memory. The .NET assemblies would write JIT compiled instructions into heap and execude code from there and I wanted to do something similar to windbg to find out if this was really .NET code or just exploit shellcode.
So, I attached a debugger to the debugger to get an understanding of what was going on. And it worked. There were mscorlib methods that could be called to convert a heap address to corresponding .NET symbol. If no symbol was found, we would raise our bar of suspicion.
The idea was found woth patenting when I brought this up, but couldn’t make it to the final list of patent applications that year. I then ported the same code to x64 version of .NET and researching this project involved downloading and going through some of the .NET framework code. The worst part of working with .NET being you end up starting a lot of your sentences with a dot!

7. Driverless, API hooking-less process, network, filesystem and registry activity monitoring.
This was in 2018. The idea came from Linux which has a very credible auditing framework. We did not want to be inline and just wanted to collect enough data to detect. I clearly remember wondering how it would work in Windows when one of my colleagues described the audit-based solution in Linux. Having come from the world of exploit detection and API hooks and drivers, this was a new kind of challenge.
Turned out Windows did have as much of an advanced user-mode ‘auditing’ system as linux in form of ‘Kernel mode’ ETW events. And there was a WMI way to get those events too.
(I still wonder about the mysteries of WMI! The classes and object-model of an entire framework exposed to powershell and WQL, is like an untapped source of most potent hacks!)
A lot of my friends in Windows cybersecurity development community had heard about ETW based events but never used them. There was not enough documentation and of course no source code to copy from. The events were in binary format and MS documented the events’ structure much later after I had already parsed them. In most cases I just needed the path names which were available at a fixed offset from event start address when viewed from a debugger.
The biggest challenge here was registry events. There is no documentation about event types, how the events are related to each other and what each event meant. The challenge here was from not knowing and not being able to google. We just had to figure everything out by trial and error.
At the end of the day we had our big learning that if you are just interested in auditd like passive monitoring, you do not need a driver and API hooks on Windows. Neat.

8. COM API interception to detect what shell and its extensions were doing
This was back in 2010 when Windows 7 was the cool new thing. Window explorer did not call CopyFile API anymore throwing our DLP detection logic off track. There were just shell APIs getting called when you moved or copied a file, and they would just WriteFile to destination. We wanted the source file too so we could classify for potential sensitive content.
API hooking is a much-discussed topic, and I was a follower of many techniques in use back the day. None of them covered intercepting inter module calls to COM methods. One technique was to code your own implementation of the particular COM interface and hook QueryInterface to return an instance of your class and call the underlying OS class from within yours. This worked just-fine except that it required to implement the entire class.
We knew that internally method calls were implemented as API calls with self pointer as first argument so we could just intercept any method we want as API. Problem was that those APIs were not exported addresses and so we just could not GetProcAddress them. But we could get to the addresses using offset calculation from instance pointers and needed a way to place hooks at specific addresses.
Ultimately we got our API hooking library vendor to implement hooks based on addresses and the thing worked like a charm.

9. Firefox plugins to detect unencryoted traffic.
I remember this to be around the time when 2009 recession hit us.
We needed to detect file uploads, emails and any post requests sent to https sites that could leak sensitive data. API hooking was not a friend as we did not know what API’s to hook. In came XPCOM plugins, deployment of which was hacked by editing firefox config files on disk. We installed a well known firefox plugin and observed the filesyatem changes it made. We then made the same changes for our plugin (and throw in some config options to even bypass signing requirements).
The challenge here was to find how to deploy a sink to get posted https traffic. Debuggers could not help us, we had to actually read through the firefox source code. This of course was a very hard topic to google. Finally somewhere in some obscure discussion thread about NS (Netscape) APIs, a method name showed up. I pivoted my research around that method and looked at examples of some very old plugins to arrive at a workable solution.
This worked for many versions of firefox till a firefox rewrite broke our implementation and ultimately my colleague used address based hooking to patch an internal firefox API that worked.

10. Restart a box into Linux.
This was a small project I was tasked to do in a new team that I joined in 2014 after the previous set of developers who wrote it had to leave. So I was left with new code, no documentation and code comments in Mandarin! The challenge here was this was new space. One had not worked in the mysitical world of Linux ever before.
The idea was to create an image file on disk and reboot the machine into a small OS running from that image. Checken and egg!
There were Ubuntu installers (Wubi)that could reboot your windows box into a temporary Linux image and this piece followed similar technique. While that was easy in BIOS based systems, it became harder or even impossible to achieve for UEFI boxes because the boot order was now burnt into UEFI firmware when the OS was installed on a machine.
Eventually the whole architecture was changed to use Windows Repair Environment which was much more flexible and friendly.

11. Algo to convert json into database rows
This was as recent as 2020. We wanted to export database-like tables out of Kubernetes, AWS, and any API based rest-ful interface. I always maintained that we should have generic a mechanism to convert restful API response into table, as we could not write code for all API responses in the world. People did not believe if such generic model was possible unless there was a demo.
You give a rest interface address that returns a json, you see a row and column formatted table! Cool! Magic!

12. Detect code injection and process hollowing without drivers and API hooking
This was again in 2020. We wanted to top score as a malware detection system and needed to detect some deceptive techniques used by malware. Did not have the luxury of API hooks and drivers and just depended on Windows events and ETW framework.
Finally, we found a way to detect memory allocation and de-allocation, image load and unload and thread creation by one process into another using just ETW events. Again, the thrill of this solution is in its simplicity and low-level of intrusiveness.

13. Inject into Windows 8 metro apps.
This was back in 2013, and Windows 8 metro apps were the cool new things. The Windows 10 store apps inherit the same design. Microsoft brought ‘each process is its own user’ architecture in Windows and we were finding it hard to inject DLLs into those ‘sandboxed’processes. That was till it turned out to be an ACL issue. We just saw some ACLs present in system DLLs that were loaded by these processes, which were missing from our DLL. We added those ACL entries and in we were injected into the new world of cool Metro Apps.

14. Bug in detours without source code.
In 2017 there was a bug we were tracking that was only seen on one of the QA engineer’s laptop and did not repro easily. There was as infinite loop in certain conditions in MS office applications while deploying some API hooks. We used a modified version of Microsoft detours for API hooking and the loop was somewhere within the detours code.
Then one day that issue happened on my laptop, so I dropped everything I was working on and started chasing this bug down, attached the debugger and yes, the loop was within the detours code.
I had source code of a previous version of detours and from there nothing seemed wrong. But then the assembly instructions started to diverge from source code at some place and there was one specific place where in a failure condition, the detours code was calling a different function than the one specified in code and I was able to prove without reasonable doubt that the version of detours we had is mistakngly calling a function in an error condition that was causing it to loop infinitely.
I was asked to record my findings in an email and forward it to the managers. Few weeks later I received a new version of detours with the fix. The particular library went on to become open source.

15. Bug repro in notepad
A race condition in an injected DLL was causing a crash of notepad. Bug was really really hard to repro. Never repro’d in front of me, never happened with debugger attached and QA saw that as a blocker. There was some kind of time constraint so I added an unhandled exception filter in the hook DLL and wrote the code to create the application minidump whenever there was a crash.
The issue could not repro with the newly added exception filter. That is when I asked the QA enginee to run a bat script to launch notepad.exe a 1000 times and see if any dump files were created.
There was one that was formed in the 20th attempt. We got the dump, we found the crash, we fixed it and shipped the software. They still use that script, I am told.