Yr12 Journal 21

Comments

In Term 3 Week 8, I optimized the performance of my project and I tested it.

Progress

  • I added a cache data strategy to store transaction data to local, and next time when request, it will use the local data if applicable:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    def get_history_df(coin, year, month):
      ck = []
      bk = []
      cache_path = f"saved_data/{coin}_{year}_{month}.json"
      os.makedirs(os.path.dirname(cache_path), exist_ok=True)
      # load from cache
      if exists(cache_path):
          f = open(cache_path)
          data = json.load(f)
          ck = data['ck']
          bk = data['bk']
          f.close()
      else:
          for i in range(1, 28):
              # 10hr data
              s = u.get_certain_time(year, month, i, 0, 0)
              e = u.get_certain_time(year, month, i, 10, 0)
              # coin kline
              coin_k = get_k_for_coin(coin, s, e)
              # btc kline
              btc_k = get_k_for_BTC_baseline(s, e)
              ck += coin_k
              bk += btc_k
              # 10hr data
              s = u.get_certain_time(year, month, i, 10, 0)
              e = u.get_certain_time(year, month, i, 20, 0)
              # coin kline
              coin_k = get_k_for_coin(coin, s, e)
              # btc kline
              btc_k = get_k_for_BTC_baseline(s, e)
              ck += coin_k
              bk += btc_k
              # 4hr data
              s = u.get_certain_time(year, month, i, 21, 0)
              e = u.get_certain_time(year, month, i, 23, 59)
              # coin kline
              coin_k = get_k_for_coin(coin, s, e)
              # btc kline
              btc_k = get_k_for_BTC_baseline(s, e)
              ck += coin_k
              bk += btc_k
          # save
          with open(cache_path, 'w+') as f:
              json.dump({
                  'ck': ck,
                  'bk': bk
              }, f)
      # dataframe
      df = u.get_dataframe_from_kline(ck, bk)
      return df

    It will parse a cache file path, and if the file exists, it will decode it using JSON and get the required lists, if it does not exists it will request binance’s API and gather the data, at the end store it as a JSON object to the cache path.

    • I also optimized the performance predicting the next 15 mins price:
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      @app.route('/predict_<coin>_15')
      def predict_15_min(coin):
          coin = coin.upper()
          try:
              btc = []
              aud = []
              # past data
              ck, bk, coin_k, btc_k = d.download_last_hr_data(coin)
              for i in range(1, 16):
                  b, a = get_price_min(coin, i, ck, bk, coin_k, btc_k)
                  btc.append(b)
                  aud.append(a)
              return {
                  'result': 200,
                  'btc_price': [float(b) for b in btc],
                  'aud_price': [float(a) for a in aud],
              }
          except:
              return {
                  'result': 500
              }

    The old version will download last hr data each time when it make a prediction (for each iteration), and now it only downloads once, which would be faster.

  • The optimized version runs 10,000% faster to verify a model after the relevant transaction data is cached.

  • The optimized version runs 200% faster to predict the price in the next 15 mins.

Meanwhile, I tested my project, it is kinda accurate:

Challenges

  • Nothing is particularly challenging except the pressure from AST, and I was really tired after it.

Reflection

I think I did really well this week. Although I didn’t learn much, but at least I optimised my project. I successfully made it run much faster and I am glad. We did not have much class this week, but I used my class time wisely, I released my stress in the class time, which enhances my performance. I believe releasing stress is important, and due to I basically finished everything in my Data Science Course, I chose to relax and watch film on my laptop, it was actually quite helpful and brought me heaps of inspirations. Overall it was a good and efficient week.

Timeline

I followed my timeline and successfully tested my project, it is accurate. I will keep testing it in week 9, 10 and 11. And by Week 12 I will start making my presentation and documentation.