openssl : ct_log_list.cnf

This is a tutorial about parsing a list of certificate transparency log servers (that are in OpenSSL requested formats)

If you use OpenSSL v1+, you might realize that your openssl s_client now has an option to parse CT logs from a certificate.

Some information about those flags:

-ct Request and parse SCTs (also enables OCSP stapling)
-noct Do not request or parse SCTs (default)
-ctlogfile infile CT log list CONF file

Basically, that’s a Certificate Transparency log that (mostly) CA embedded into your certificate, which is required by Chrome to not throw an error when you visit a site that has a certificate issued by this CA.

When you run the program with openssl s_client -ct the first time (and every time you don’t have a CT log configuration file), you would (normally) expect an error like this

12580:error:02001002:system library:fopen:No such file or directory:crypto\bio\bss_file.c:74:fopen('C:\Program Files\Common Files\SSL/ct_log_list.cnf','rb')
12580:error:2006D080:BIO routines:BIO_new_file:no such file:crypto\bio\bss_file.c:81:
12580:error:0E078072:configuration file routines:def_load:no such file:crypto\conf\conf_def.c:150:
12580:error:3207B06D:CT routines:CTLOG_STORE_load_file:log conf invalid:crypto\ct\ct_log.c:209: 
This is from a windows device.

Now, how to resolve this issue? (or, what is a ct_log_list.cnf file and how could i get one?)

If you dig on google.com, you would found those (somewhat unhelpful) lines: Link

# This file specifies the Certificate Transparency logs
# that are to be trusted.

# Google's list of logs can be found here:
# www.certificate-transparency.org/known-logs
# A Python program to convert the log list to OpenSSL's format can be
# found here:
# https://github.com/google/certificate-transparency/blob/master/python/utilities/log_list/print_log_list.py
# Use the "--openssl_output" flag.

Useless, hgh?

Some talent users.. (not me) might figure out how to do it…. or like me… stuck at one error forever.. and struggle to find the answer.

Let’s find a way to parse the log.. (If you don’t want to see the steps, scroll down to the bottom & I have one .cnf file parsed)

Prerequisites:

  1. have python, wget , with command line cli (and could execute py files in command line)
  2. familiar with command line
  3. could install dependencies by yourself
  4. Linux Machine (or please use the parsed ct_log_list.cnf on the end of this tutorial)

Steps:

1. Go to the link.

You’ll need to see Google’s official repo before you download it… (make sure it’s still valid & exist)

Actually, go to this link https://github.com/google/certificate-transparency/tree/master/python/utilities/log_list since you would need to download all files from this folder.

2. Make a directory at a local machine (preferably Linux..) & download all ct tools from Google (Github repo)

Go to your machine, create one folder (name whatever you want)

Download all files in this folder

Commands to execute (Step 2)

cd ~/    #move to your home folder
mkdir ct_logs      #make a directory called ct_logs, under your userhome
cd ~/ct_logs/
# Download all necessary python files from google ct tools
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/print_log_list.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/cpp_generator.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/java_generator.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/openssl_generator.py

3. Download ct_lists from Google (certificate transparency)

First, visit http://www.certificate-transparency.org/known-logs, which is a list of all logs chrome currently have & added

When you run print_all_log.py, it’s going to ask for several files, else it would throw an error.

List of files needed:

  1. log_list.json (with flag –log_list=log_list.json) (This is the file that contains all log servers included in chrome) or all_logs_list.json (with flag –log_list=all_logs_list.json) (all ct servers announced)
  2. log_list_pubkey.pem(with flag –signer_key=log_list_pubkey.pem) (Signature file, log_list.json ‘s signer public key)
  3. log_list.sig (with flag –signature=log_list.sig) (Signature file, prove that log_list is not modified)

Now, we could download all three files from http://www.certificate-transparency.org/known-logs.

Download log_list.json with this url: https://www.gstatic.com/ct/log_list/log_list.json , log_list.sig from https://www.gstatic.com/ct/log_list/log_list.sig , log_list_pubkey.pem from https://www.gstatic.com/ct/log_list/log_list_pubkey.pem

Command to execute (Step 3)

cd ~/ct_logs/
# Download all necessary ct log & signature files from google
wget https://www.gstatic.com/ct/log_list/log_list.json
wget https://www.gstatic.com/ct/log_list/log_list.sig
wget https://www.gstatic.com/ct/log_list/log_list_pubkey.pem
wget https://www.gstatic.com/ct/log_list/all_logs_list.json
# also need to download a schema in order to parse the log server file correctly, into data folder
mkdir data
cd data
wget https://www.gstatic.com/ct/log_list/log_list_schema.json

4. install all dependencies

Now you could run the python file

cd ~/ct_logs/
python print_log_list.py

You might see this error output:

Traceback (most recent call last):
  File "print_log_list.py", line 11, in <module>
    from absl import flags as gflags
ImportError: No module named absl

This means absl package is missing, you would install it from pip, if you don’t have pip and you don’t wish to install pip, compile absl manually from source https://github.com/abseil/abseil-py

pip install absl-py

Now run the script again, you would see another error

Traceback (most recent call last):
  File "print_log_list.py", line 12, in <module>
    import jsonschema
ImportError: No module named jsonschema

This means jsonschema package is missing, you would install it from pip

sudo pip install jsonschema

Run the script, yet another error

Traceback (most recent call last):
  File "print_log_list.py", line 13, in <module>
    import M2Crypto
ImportError: No module named M2Crypto

Install m2crypto from pip

sudo pip install m2crypto

After this, the script should be running, (but throw error like the following ones)

Traceback (most recent call last):
  File "print_log_list.py", line 115, in <module>
    sys.argv = FLAGS(sys.argv)
  File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 633, in __call__
    self._assert_all_validators()
  File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 507, in _assert_all_validators
    self._assert_validators(all_validators)
  File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 528, in _assert_validators
    raise _exceptions.IllegalFlagValueError('%s: %s' % (message, str(e)))
absl.flags._exceptions.IllegalFlagValueError: flag --log_list=None: Flag --log_list must be specified.

This means you are on the path, proceed to the next step

5. parse the log file

This would require a few flags with files, we’ve downloaded all in the previous steps.

Now, if you want to use the CT log server that are in Chrome, run the below command, replace ‘ooo‘ (after –openssl_output) with your OpenSSL ct config file path, or where you want the ct_log_list.cnf be placed

python print_log_list.py --log_list log_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo

Now, if you want to use all announced CT log server, run the below command, replace ‘ooo‘ (after –openssl_output) with your OpenSSL ct config file path, or where you want the ct_log_list.cnf be placed

python print_log_list.py --log_list all_logs_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo

If no errors are present, here you go..

You are done, open the file & place it to where your openssl wants

Here’s the ct_log_list (generated on 9/3/2018), download it & rename the extension to cnf or conf, depend on what the openssl wants.

Have questions or feedback? Leave it in comment area below

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.