Key highlights of the sessions and hacking villages that I attended at the 2020 BSidesSF conference in San Francisco. This year the conversation inspired discussions around changing the way in which we work in cybersecurity, by getting people more involved or approaching the same problems in a different, more efficient way. Read on to find out what the SF cybersecurity scene had to say.

I thoroughly enjoyed my first BSidesSF conference, a volunteer-run engineering-oriented security conference which deals with real problems that security teams face day in and day out. As the name implies the BSides conference is always paired with a more mainstream security conference (in this case it happens on the two days before the RSA Conference). I had high expectations after attending BSides London a few years back in the run-up to 44Con, and the San Francisco version did not let me down.

Sessions

Sessions were one of the main reasons why I attended the conference as it provides a wealth of knowledge of what other organisations security teams are facing without the vendor pitch. Below is a rendition in chronological order of the chats I attended - there were up to four tracks running in parallel so this is by no means a representation of everything that went on at BSides. You can find recordings of these sessions and all the other ones here.

Keynote for Day One: Give Away Security’s Legos: Dumping Traditional Security Teams by @fredrickl

The keynote for the first day was a fresh approach to the “you are doing it wrong” theme: move away from the bureaucracy that your security organisation may be trapped in by devolving security to the departments you usually work with, let them take ownership, and allow the team to focus on doing real cybersecurity work that no-one else at the company can do: threat hunting, developing security components and consulting on security topics.

The presentation was full of pragmatic approaches on how to do this, here are some of my personal takeaways (but I strongly recommend you watch the recording of the presentation, Flee has a lot to say):

• You do not have the full context of your business: during security reviews we spend a lot more time trying to familiarise ourselves with the proposed change and the business it supports than actually conducting the security review. While I agree that we cannot be in the middle of every single change, there still needs to be exposure of the cybersecurity team to the business, we need to be able to understand our organisation challenges and do security that makes sense and is aligned with the company strategy.

• Product manager involvement: moving the responsibility for threat assessments and security reviews to Product Management, making security part of their product (like we always preach) but letting them do the risk assessments and discuss with the Product Owners the prioritisation (something I never do). But see my point above, we still need to be close to Product, just not in the middle of it.

• Awareness training: moving away from the generic “policy” mandatory training and doing role-focused training, giving each team the knowledge and tools they know to do security - not just to follow the rules. Train them on how to do threat analysis, risk assessments, manage vulnerabilities and how to use the tools that the organisation has invested in.

• Table top exercises: we have done paper exercises to simulate attacks and crisis management, however modelling after a D&D game may be the way to go to get people really engaged and after all, as Flee says, everyone loves D&D - even if they do not yet know it :-)

• Security people are not the smartest. Security people get it wrong: we seem to be at the centre of the universe, hiding behind the disguise of consultants and advisors when we really are the gatekeepers/cops. People humour us as a necessary evil, but really are happy to see our back and get on with their work. We are neither the smarter kids on the block and can also do mistakes, so stop pretending - letting the other teams do security makes the organisation take better security decisions.

Having the risk ownership with to the business owners, where it really resides, allows the right people to make the right decisions when it comes to trade-offs - supported whenever necessary by the security team. We should spend less time trying to get our fingers in all the pies, pretending we are the centre of the business and focusing on delivering the outcomes we are uniquely placed to deliver.

Graph Based Detection and Response with Grapl by @insanitybit

Although some of the security products we are currently using are already incorporating graph-based analysis (endpoint protection, network anomaly detection or auditing - like the speaker-referenced BloodHoundAD come to mind), I thought that Grapl, the open-source tool that the speaker has developed, was a novel way of looking at attack detection beyond traditional log analysis.

The concept is simple and powerful, which typically a good combination: you feed logs to the tool in real-time and it applies a set of graph-based models to detect when something is not as it should be, not that different to how ESP works.

The models are defined in python, which means you do not need to learn an additional language and that you can apply all software engineering best practices to it (repositories, versioning, sharing, testing…).

GRAPL parses logs using a set of connectors (sysmon and a json-specific format) and converts this into a graph database, adding nodes and edges as each log provides additional information to an existing process, network connection, user, etc. This means that your detection rules no longer need to operate on a single log-line (or even the context of the past 30minutes of log-lines) but on an entity that evolves and that is enriched by every log line that is received. This makes it really simple something that is difficult for SIEMs without using unapproachable amount of memory: matching logs from 2/3 different sources over a period beyond a few hours.

This graph database is then analysed using a set of rules, such as “identify all nodes started by Word/Adobe/Excel/Outlook that have created subprocesses that have connected to an external IP address and downloaded a new file that was then subsequently ran” or “identify all services that are exposed externally and that have executed a subprocess less than 5 times in the past month”, etc. I can see the value of sharing these rule-sets which are technology-agnostic and really try to model process and user behaviour, something similar to what Microsoft is doing with Sentinel play-books.

The concept of “lenses” was also introduced as a way of exploring and doing forensics by starting at a node and moving through it to other nodes.

I can’t really wait to give Grapl a try!

The speaker hinted at interest from commercial vendors for the solution, so I guess the advice is to fork it while it lasts :-)

Secure by Design: Usable Security Tooling by @hxnyk

An interesting, although brief, discussion on how improving the usability of security tools leads to increased adoption amongst your internal customers.

As a security engineer I tend to focus a lot on the core functionality of the application I am working on, e.g. what makes it different from anything else out there. I spend a lot of time on that core functionality, testing, being amazed at how awesome the research results are and celebrating. It is only at that point that I just patch together a UI around it, as quickly as I can, before I release it to the world.

It is usually at that stage when the application fails.

Hon’s talk starts with a typical security engineer designed UI, pointing at several common and easy to overcome flaws such as confusing controls (my favourite sin), lack of intuitiveness, no deep-linking ability, etc. All of these flows make the application intimidating and, to quote Han “Instead of learning security the user need to learn about your tool before they can start learning security”.

While these are easy to avoid flaws, the secret is on putting yourselves in the user shoes and using the work that already exists out there: if your company does public-facing products involve your UX designers in the development of your tool, sit down with users and listen to them and apply usability best practices such as Nielsen’s heuristics.

While I agree with most of the arguments in favour of the importance of a correctly designed UI to improve user engagement, there is one concept that I cannot get myself to agree on “UI before API”, e.g. designing the API around the UI instead of the other way around. To me an API needs to be created around the business domain and the models your application handles more than with a specific UI in mind. APIs exist only for a purpose: separating the frontend from the business logic concerns, so that any number of UIs can be created without having to modify the back-end. I cannot really see any API that I design be driven for how the UI will represent the information!

The Red Square: Mapping the Connections Inside Russia’s APT Ecosystem by @arieitan

A brief presentation by Ari Eitan at Intezer regarding an analysis of several APTs attributed to Russia state-sponsored groups, breaking down its components and trying to establish patterns amongst different apparently independent criminal groups. The findings proved that there was no technical link between the various APT groups, however whether this was by design or as a form of misdirection was left as an exercise for the reader. Of the various presentations this is the one that felt the most commercial, with the work based on proprietary technology only available to Intezer, so difficult to validate. They presented a couple of tools that were the result of the study, both available at apt-ecosystem.com.

Peeling the Web Application Security Onion Without Tears by Noam Lorberbaum and Keith Mashinter

A detailed view into how Adobe has re-architected Adobe Document Cloud and how they have leveraged existing security technologies to provide a defence in-depth approach, without reinventing the wheel. It was good to see how similar challenges are resolved in similar ways by different organisations, from providing DDoS protection layers to addressing the security of API gateways.

In an effort that would not be alien to any organisation that has to comply with multiple regulations across different jurisdictions, Adobe has created a common control framework and open-sourced it so that other organisations can put it in place - good reading to any organisation that operates any customer-facing product.

Keynote for Day Two: What’s New or Not in 2020: Are we Making Progress on the Intractable Security Problems? by @larkinryder

A review of what major shifts in cybersecurity over the past decade, presented in an entertaining way so make sure you check the whole video. Some of my key takeaways:

• Third-parry trust: we are doing it too manually and too “point in time”, we need a better way to manage the trust of third-parties as we increasingly rely on them in our organisations

• Privacy: an enabler to do security right over the past few years, take GDPR or the CCPA as a way to ensure that organisations are doing the right thing. A point that, without disagreeing, leaves many things unsaid on the values of those organisations that are only driven by compliance when it comes to the security of their customers and stakeholders.

• Cloud cloud cloud: a common theme across the conference, everyone is shifting loads to the cloud and it is becoming rarer to find organisations that manage their own infrastructure. There were a couple of very visual slides on how the skill-set of Technology teams have moved a couple of layers up and is not focused on Applications. Through the conference and no matter who you spoke with it was difficult to find anyone that is running on-premises systems any more.

• Mobile ubiquity is good for security: the push to be able to connect to all our data from wherever we are and through our mobile devices has opened opportunities to increase security authentication using mobile built-in 2FA mechanisms or even biometrics. We should leverage the use of devices to better identify our customers. On the other side losing mobile devices has become a real problem, encryption is a must if you do not want to see your data stolen.

• Customer data is off limits: long gone are the days where customer data was passed around between teams with disregard on user information security. Nowadays it is difficult to demonstrate why a team needs access to personal data, creating guard-rails to ensure that data access is appropriately protected and audited goes a long way to avoid breaches on that information. One very interesting though that was presented is trying to understand how much criminals resell breached information for so that we understand what is an appropriate investment (e.g. if bad guys are selling passport information for USD1,000 then it makes sense we invest a similar amount of money to protect them).

Protecting the Bridge from Dollars to Bitcoin: Securing Coinbases’s Edge Payments Infrastructure by Nishil Shah

A deep-dive on how payment methods work in the US (and abroad) and how difficult it is to protect loses with the controls built into payment systems that have been around for decades, well before trust between the different entities that participate in a transaction was an issue. A threat analysis of various payment method flows was presented with call-out to understand what could go wrong and how companies address it.

Coinbase integrates lots of different payment methods, some of which are more creative than others - to ensure security on those payment methods a three-step process is followed:

• Review the new payment provider during the procurement phase, and stop it at that stage if it does not provide appropriate assurances

• Regularly test the payment integrations

• Incorporate security test cases in the integrations developed, by having security champions embedded in the payments engineering teams

One of the interesting thoughts in Nishil’s presentation was whether a payment bug that led to money being lost was really an application security topic (something we can also extend to other items that contribute to fraud). The bottom line is that if we define security as ensuring that the amount of fraud is prevented then it falls spot on in our area, so do not shy away and work with those teams - even if it is not strictly your responsibility: in the end we are all trying to make our organisation as successful as possible, and preventing fraud sounds like a great way to contribute to that goal.

Lessons Learned from the DevSecOps Trenches

A panel of application security leads, CISOs, Red Teams leads and security researchers that have been working in DevSecOps for a while and that shared some of their success stories and pitfalls to avoid:

@astha_singhal, Director of Application Security, Netflix
@dugdep, Director of Defense, Datadog
@justine_osborne, Offensive Security Technical Lead, Apple
@zanelackey, Chief Security Officer, Signal Sciences
@clintgiblertldrsec.com, Research Director, NCC Group

The panel addressed topics in various areas which I have tried to capture below, however it is probably better if you watch the panel itself and form your own conclusions.

Failure stories

• Testing matters - think about the actual operation of the tool you are developing before you release on your production environment
• Set clear success criteria before you spend a year on projects that are going nowhere

Asset inventory: if you were to start embarking on a path from scratch how would you approach it?

• Gotcha: realise that you do not need just security engineers, but data engineering analysts so that the tool scales
• Positive: organisational side, send security champions to meetings with the product and build infrastructure team to learn and listen on new products that are coming up

Build vs Buy decisions

• The key decision point are the resources that need to be dedicated to development and maintenance
• Based on the vendor you are working with, getting 50% immediately from a vendor may be better than 100% in a few years time!
• Get 80% coverage as quick as possible, so that when the resources are available they can go into the remaining 20%
• Be open to deprecating internally developed tools once the vendors catch-up

Security analysis in pipelines in CI/CD

• “Trying to find all the bugs” vs “Are we using the frameworks in use”? It is better to build guardrails for developers than trying to find complex issues (e.g. use mutual-TLS between services, use secret management, etc.)
• A lot of critical vulnerabilities found at Netflix could have been resolved by implementing best practices, so they focused on the controls, e.g. not avoiding the vulnerabilities but limiting the impact.
• ..but it’s not always about using security components to get security but default but doing security and understanding unique risks as well.

Red Teams: how do you make sure it is improving the security posture

• Measuring security risk is a hard problem, but security testing boils down to hypothesis, experiment and analyse the results. Defining the correct hypothesis for the organisation is the key.
• Red Teams allow to see the bigger picture: there are always loopholes, etc. It also allows to move beyond jut AppSec and find issues in other areas (infra, corp)
• The impact is not just the number of bugs found, but the new types of attacks you are identifying as a blue team
• Do not call it done until all the things you found are no longer possible to be exploited
• Red Teams can also be a good tool, but you also need to balance that by not breaking trust with internal customers (e.g. so you are not perceived as showing how dumb they are by breaking into their systems)

Threat modelling and security reviews: how important and how do you scale?

• Use MITRE ATT&CK framework as the basis for thread modelling, so that the whole detection is improved
• Scaling thread model is a requirement: self-service questionnaires for developers (so that the output recommends specific guidance on what to do and it computes a risk score, so the dev security can focus on the riskiest), or integrating threat modelling on the development life-cycle or thread-model as code (e.g. adding security tests as abuse in integration testing)

AppSec pipelines: What static/dynamic analysis has brought the highest ROI?

• Really targeted scanning, e.g. instead of doing everything well focus on a class of bugs and kill it everywhere
• Start bottoms-up and define initial security controls, so that the development team self-manages the issues and their resolution.
• Focus on dangerous patterns that are specific to the frameworks in use
• Identify as quickly as possible, e.g. on code commit or in the IDE - or at the worst case in CI/CD pipeline
• Finding vulnerable dependencies in close partnership with build teams (they have the same challenges to manage dependencies) so that we can flag dependencies

What is the most effective engagement and awareness initiative you have done with your engineering teams?

• Find ways to have engineering teams engaged with security, so that security is centrally located or balls of candy on people coming to central team. Budget for drinks on engineering teams happy hour last two rounds :-)
• Asking Engineering to join the Red Team the exploitation of a vulnerability
• Create a good experience when folks come to talk to you, e.g. be realistic on the goals and be as responsible as possible

New organisation: what would be the first investment you will do?

• 2 x Visibility into what you have, what it is and who owns it - e.g. visibility on your systems
• Protect a number of security analysts from interruptions so they can take the long-view on some hard problems (e.g. framework creation, etc.)
• Make sure you understand the risk you are trying to reduce at the enterprise level (e.g. what are the corporate nightmares when it comes to risk) so you can prioritise
• Going into a new security roles: get the buy-in from the other teams, they for sure had negative experiences in the past, so you need to change that perception for future successes
• Vulnerability management: so you can measure when something worked or not

2FA in 2020 and beyond by @kelleyrobinson

This was a really useful session with Kelley presenting research on the usability and effectiveness of various two-factor authentication mechanisms, covering SMS, TOTP, Push and Physical security keys. I believe that a quote from the speaker “We are so owned passwords are no longer safe” mimics reality, for all our important accounts we are demanding further authentication mechanisms, so I found the talk very interesting - specially if you are deciding what next 2FA mechanism to deploy. I recommend you watch the whole presentation as it covers a lot of useful statistics around 2FA methods penetration and usability.

Some of the results of the research were learnings for me, specifically the following two:

• Push is not as effective as we think: It is probably the easiest and less-friction mechanism for 2FA, but this seamlessness makes users feel that the system is not adding that much security. It does not help that there is no standard around Push security approvals and that everyone needs to install its own app for this.

• TOTP has a wider adoption than thought: It combines the security that SMS cannot offer, the ability to work offline and it only requires the user to install an app - which they have already done if they are using YOUR app. Entering the codes gives the right amount of security perception that customers are looking for, and it uses open standards so there are multiple ways of getting the codes.

OTR: Tears from The Cloud by @theckman

BSidesSF has introduced a new type of sessions this year labelled OTR (“Off the Record”) where any recording of the session or discussion on what was mentioned is discouraged, in a concept similar to the DEFCON Sky Talks.

These sesssions are out of bounds for the press, with the goal of having open and frank discussions without the risk of being shot down for bringing up sensitive topics. I attended this particular session and, while I cannot really discuss what was mentioned, I think that the OTR sessions are still far from perfect both in content and interest - and from what I heard from people in my team that attended DEFCON this year the same can be said for the Sky Talks.

Mental Health for Hackers: Contents under pressure by @v33na, @ryanlouie and @ChloeMessdaghi

This was the final session I attended at BSidesSF this year, I wanted to understand the specific mental problems that people in Cybersecurity face so I can be proactive in identifying, thinking of team sanity more than anything else. While the session was useful on getting some background information it lacked specifics on how employment in a cyber position contributes to these mental health issues. We know people in our space are affected by Anxiety, Depression or Burnout - but what is it in our job that makes these problems so prominent and what can we do about it?

One of the key points I took away was the concept that all of these problems are easier to handle if you share it with someone, if you do not go through it alone - it’s the community that matters and both in cyber and security research there is a large amount of those. So do not do it alone :-)

The closing notes of the presentation gave some pointers for further research:

• American Psychiatric Association

• National Institute of National Health

• World Health Organization

• Mental Health Hackers @HackersHealth

Villages

There are five “hacking” villages where you can learn the basics around tinkering with various areas that require some basic hardware you may not have around your workshop, using a self-learning tutorial style - just sit down and get started:

• IoT: a very introductory tutorial around firmware analysis, spotting hard-coded credentials and exploiting them through telnet. Learnt about binwalk as a tool to identify known structures in a packed firmware (for instance the bootloader, a squash FS, etc.) and extracting them, really cool and something that goes beyond my “strings” skills.

• Reverse Engineering: a focus on reverse-engineering firmware from a Buffalo NAS server, identifying and exploiting vulnerabilities. I learnt a couple of tricks such as the uncompyle6 tool for decompiling .pyc to an approximation of the original Python source code. Throughout the tutorial

• Car hacking: Simulating a set of vehicle components, the lab focused around CAN bus hacking, which is an area we had touched in the past when doing SCADA security research with @xpanadero for S4. Unfortunately the environment was being reset when I sat down so did not get to learn anything new.

The conference was packed with sessions so I was unable to sit down at the Lockpicking or the Crypto hacking villages.

Vendors

I did not spend that much time visiting vendors booths as I will have plenty of opportunities at RSA during the rest of the week, but there were three that caught my eye:

• >_cmd: a command-line auditing and blocking solution, similar to what Balabit are doing with PSM, but without the RDP or file transfer support. Instead of installing a gateway before accessing the system. _cmd is installing as an agent on each Linux system (it only supports Linux) which injects a library into each process which (I assume) patches syscalls so that it can monitor what the application is doing. It sends the recorded events (input/output/UID change/etc.) to a cloud-based central repository that can be used for auditing. Activities can also be blocked from happening. It feels like a good solution, but given the limited platform availability I am not sure if we are not better off with auditd for the time being. The cloud-based console is cool though, and it makes it easy to review sessions but, again, only for Linux based systems.

• snykj: an open source vulnerability tracking outfit that helps identify known vulnerabilities that you are importing into your code through dependencies, similar to what WhiteSource or BlackDuck are doing.

• nightfall AI: a cloud-based solution that monitors your information and automatically classifies based on its content and certain AI models, allowing you to apply DLP-like policies to it. It has connectors for several cloud-based services, however does not support on-premises systems, which limits its use in environments where some of the critical assets are still maintained internally, think legacy databases.

Closing thoughts

All in all a very enjoyable and useful conference, created by and for the community and for a meagre USD50 that covers sessions, villages, free swag, stickers, food, drinks and a lot of fun. There seems to be a shift in focus from extremely technical sessions towards a more “how do we improve security by looking at the human side” in general, something that I also feeling as RSAC2020 gets started (this years’s motto is “Human Element”). Let’s see where this leads from here, if you have any feedback/correction or further thoughts please reach out on twitter (@lluismh).