Thursday, August 9, 2012

Experts and Expertise

Cindy Murphy wrote an excellent blog post over at the CDFS (Consortium for Digital Forensics Specialists) blog titled, "Experts and Expertise" on the subject of experts and expertise.  She related her recent experience in a trial of a woman for Homicide by Negligent Usage of a Motor Vehicle.

One of the key points that stood out for me was her comment about the defense expert's testimony in court, " He made this determination because he observed an out of sequence SMS text record with ‘mysterious’ extra content following the message."

In the world of digital forensics, the word "mysterious" should never be used in the context of digital data, no matter where is comes from.  If he did not know the reason for the data being present and the source for that data, then testifying about the data is going to be and was in this case, problematic.

While it is okay, in some limited circumstances for an expert to say they don't know something, the practice should be to always know the answer when it comes to evidence examined and reported on in a forensic report.

One of the things I emphasize at all times with my examiners is that they better know the correct answer for any thing that put in a report.

What I mean is this:  Writing an expert report for the purpose of testimony requires that the examiner know what every item in the report means and that they have the ability to explain it in a way a trier of fact can understand.

I have read hundreds of expert reports from examiners in civil and criminal cases.  One of the most perplexing things to me is an expert who gets on the stand and cannot answer the questions about the information they put in their own report.

Perhaps there was a time when an expert could expect that their report would not be challenged by someone who does know the answer.  That situation is still all too common where an expert can get on the stand and go unchallenged as to the content of their report, whether they are presenting a "fact" report or an "expert" report.

One of the issues is that many experts do not understand the difference between a fact statement and an opinion.  And that their report is not necessarily their opinion.  Those of us to work in the digital forensics field on a daily basis know all about meta data, or data about data, but we rarely think about meta facts, or facts about facts.  Some times, in an effort to explain things in their report, they inadvertently express opinions rather than state facts and an explanation of those facts, or "facts about facts".

It is a difficult line to understand and requires that the expert be cognizant of the fact that every word in an expert report is subject to cross examination, and in some cases, that cross examination is going to be guided by an expert with a deeper knowledge of the subject matter than they currently possess.

This appears to be the case in the testimony that Cindy shared in her blog post.

Cindy goes on to talk about lessons learned in this case,

"What are the lessons here? Digital forensics is a specialized field and mobile device forensics is an even more specialized subfield. It is a field in a state of constant forward change and complex content. No expert, whatever their level of knowledge, longevity, experience, specialized expertise or training, should ever assume those qualities truly prepare them for court—or even a case."

I would clarify Cindy's statement by saying that preparation for court, regardless of your previous experience or training as Cindy points out, should keep you from deeply investigating every fact in your report to make sure you absolutely understand it and can explain it to the triers of fact.

Cindy also writes in her blog post about some of the things that can be done to make this less of an issue in the future.

I would add to her comments the following:

While we see the results of not properly preparing for court in this particular case, there are some root cause issues that must be addressed to make this less of an occurrence.  Enumerating and addressing those root cause issues are an important part of making the field of digital forensics less of a mine field for attorneys and clients alike.

1. Selecting the right expert.  While this seems like a "no brainer" is it probably one of the biggest issues in the field when it comes to an attorney or client having an expert who can actually help them. 

But, selecting just the right expert is not always an option for an attorney or client for the following reasons:

  • The expert is a staff expert and the option to use a different expert is not available.
  • The expert is the only one the client can afford.  Selecting an expert based on price alone is usually a bad criteria.
  • Experts may not exercise self-deselection.  In others words, an expert who fails to stop a client from hiring them for a case outside their expertise.
  • Fundamental lack of knowledge about the selection process and how to vet an expert.
2. Keeping Secrets.  One of the interesting idiosyncrasies about the field of digital forensics is the tendency to want to keep secret "knowledge" or perceived weaknesses,  from falling into the hands of other examiners.

  • It is an interesting dichotomy that forensic experts, who are supposed to be independent in their reviews and opinions, believe that sharing knowledge with the other side might expose them to critical review in their cases if they ask for help in understanding something where the "other side" might see the question.

    If you work on a case where another expert is present, I promise you are already under critical review.  We are not advocates like an attorney, so asking a technical question should not be considered an exposure of tactics or strategy.  And to be quite frank, failing to ask the question and leaving your client exposed is not a good strategy in any case.
3. Failing to use available resources.  It is really incumbent on examiners to constantly improve their knowledge based in the digital forensics field.  Cindy made it clear and I agree that this is an ever changing environment we work in and the need to constantly seek new learning is a critical factor for all forensic examiners.  The question is, why is this not part of every examiners "to do" list.

  • Experts come in all shapes and sizes when it comes to learning.  There are those who are constantly seeking a new tidbit to add to their arsenal of knowledge.  And then there are those who passively wait for the next training session so they can sit in a classroom and be fed information.  It this an indictment of people because of their approach to gaining new knowledge. Partially yes, but in general, no.  And here are the reasons why:
    • People having different learning styles.  Some people learn from reading and study in books, some people learn by asking questions on forums, and some learn best in a classroom setting.  I would not pick out any of those styles as being better or worse than another.  But if you understand your learning style, and your organization understands your learning style, maybe a plan can be created that will best benefit you and the organization.
    • Some people just don't have the motivation to seek learning due to any number of factors personal to them.  Perhaps they are burnt out, perhaps they took the job because no one else wanted it and it is not their passion. 
    • And some people simply do not know how to figure out what they need to learn, i.e. without a mentor, it can be difficult to develop a plan to augment skills because you may simply not know what you don't know.
  • The missing link, networking.  I hear this most often from my law enforcement colleagues and from solo practitioners, that they don't have someone they stay in contact with who can help them with questions or who can spend time with them to share knowledge.  Being the lone ranger can make it hard to now only gain new skills, it also makes it hard when you don't have a go to person or a mentor you can call on.  What are some possible solutions to this?
    • Join organizations that exist to share knowledge, like CDFS.
    •  Work within your particular area to establish a mentoring program.   In my humble opinion, this would be a great asset for law enforcement examiners to have a mentoring program where they would have someone to call on and to guide them along their professional path.
4. Failing to critically vet training.  There are a lot of organizations that provide training for the field, some better than others. 
  • I think it would be of great value for training organizations to ask veteran examiners to come and attend their classes from time to time and give feedback on the functional, real world impact of the training class.  Anyone providing training should be on a constant quest to continuously improve the content and relevancy of their training programs.
  • Some of the best feedback a training program could have would be to have examiners who have used the training to prepare for and testify in court where they have had an opposing expert who challenged their results.  This would be a good litmus test for how well the training served the expert in preparing for that case.
5. Failing to develop new talent for the field.
  • I think if you ask very many people trying to enter this field that they are stymied in many cases by either the lack of a certification or the lack of experience as a barrier to entry.  To be true, both of these are valid barriers to entry into the field if one is expecting to be a full fledged, case handling, independent examiner.  How could this be addressed?
    • Internships with organizations who can apprentice these newcomers to the field to prepare them for entry level positions.
    • Internal internships in law enforcement for examiners new to the position who are getting trained, but are not given the opportunity to work under experienced examiners for a period of time to get the benefit of the "road time" the experienced examiner has.  They don't put rookies on patrol day one out of the academy and new examiners should be considered "rookies" in their own way.  At the very least, the new examiner should be connected to a mentor.

This post is already way too long, so I will stop here and say that there are numerous other areas that could be addressed to improve the overall expertise in the field.  I am not saying I have all or any of the answers, and I am glad that there are groups out there like CDFS dedicated to this purpose. 

For me, I am always asking "What's missing."  And I am sad to say that what's missing at this point is a comprehensive action plan by an organization or group to address the issues brought up by Cindy and hopefully, in this blog post.

No comments:

Post a Comment

I have moderated my comments due to spam.