Merged Blocks

Reading time: 7 min

How to process payments on Shibarium Network on Node.js

Shibarium Logo

So, you have your project, and you're eager to start receiving those sweet little tokens, huh?

The first thing to know is that the Shibarium Layer 2 Blockchain is a fork of the Polygon project (according to some sources on their Discord, as the source code is still private). It operates on the EVM. All EVM blockchains use the same API for compatibility, although there might be some custom endpoints on paid API providers.

While researching how to accept payments or receive tokens, I discovered there are actually several ways to achieve this. In this article, we will review each method and analyse their adtanvages and disadvantages. If you wish to skip ahead to the selected solution click here.

There are a multitude of reasons why a user would transfer tokens to your DApp, but some of the most common are: 1) you are selling goods, and 2) your user has a balance and wants to deposit (as seen in games). In my case I want my users to have a balance and be able to deposit and withdraw, but it shouldn't be too complex to adapt these ideas to a different use case.

Note that for all examples the client need to add shibarium network to their Metamask first.

Let the client notify you when the payment is done

Diagram

As shown in the diagram this method depends on the server creating and tracking invoices and the client to be able to report back the transaction id to the server after the transaction is confirmed.

The problem with this method is that the client App is responsible to report back the Tx Id to the server. Which means that our App must be connected to the user's wallet like in Metamask. If the user wants to use a different wallet to send us tokens, our App won't have access to the transaction Id.

On the other hand if the app actually makes the transaction and then there is an error, a system to "recover" transfers should be implemented.

Pros:

  • Simple server implementation
  • Small amount of calls to the Shibarium RPC endpoint
  • Works for both payments and deposits

Cons:

  • Only works by connecting Metamask
  • Only works in Web
  • A system to recover from errors should be implemented

I've seen this implementation on several payment gateways, but for me the disadvantage of just being able to use metamask on web was the no go.

Monitor transaction count on addresses

This idea suggested by chatGPT also seems interesting.

const Web3 = require('web3');

// Replace 'YOUR_NODE_URL' with the URL of your Ethereum node or Infura endpoint
const web3 = new Web3('YOUR_NODE_URL');

// Replace 'ADDRESSES_TO_MONITOR' with an array of Ethereum addresses you want to monitor
const addressesToMonitor = ['0x123...', '0x456...', '0x789...'];

// Function to check for deposits
async function checkForDeposits() {
  try {
    // Loop through each address to check for transactions
    for (const address of addressesToMonitor) {
      const transactionCount = await web3.eth.getTransactionCount(address);

      // If there are new transactions since the last check
      if (transactionCount > 0) {
        console.log(`Deposits found for address ${address}`);

        // You can further process the transactions or trigger additional actions here

        // Update the last checked transaction count
        // (This is a simple example; in a production environment, you may want to store this information persistently)
        // lastCheckedTransactions[address] = transactionCount;
      }
    }
  } catch (error) {
    console.error('Error:', error);
  }
}

// Set an interval to periodically check for deposits (adjust the interval based on your needs)
setInterval(checkForDeposits, 60000); // Check every 1 minute

Here we would need to have the original transaction count (probably 0) of each address in our database. And track down changes on its count.

One idea that comes to mind is to monitor our unique deposit address. We would know a deposit has been made, and by checking the sender we could mark the sender's invoice as paid automatically. Thus freeing the client from reporting back the transaction Id to the server.

In the case we use different deposit addresses for each invoice. We could temporarily track pending invoices addresses until paid or timeout. This comes with the cost of implementing a timeout system to avoid tracking unpaid invoices forever tho. Another problem could be that we could end up tracking hundreds of individual addresses but you wish you had that problem xD.

In my case this wasn't a "suitable" solution since I'm using an individual deposit address for each user, so tracking every user's address every minute didn't sound very scalable to me. We are devs so we like complex solutions for imaginary scalable problems ;).

Subscribe to a Contract's events

Here we could listen for all "deposit" events on a contract or we could add filter to just listen to specific addresses.

const Web3 = require('web3');

// Replace 'YOUR_NODE_URL' with the URL of your Ethereum node or Infura endpoint
const web3 = new Web3('YOUR_NODE_URL');

// Replace 'DEPOSIT_CONTRACT_ADDRESS' with the address of the contract handling deposits
const depositContractAddress = '0xabc...';

// Replace 'DEPOSIT_EVENT_NAME' with the name of the deposit event in your contract
const depositEventName = 'Deposit';

// Optional
// Replace 'ADDRESSES_TO_MONITOR' with an array of Ethereum addresses you want to monitor
const addressesToMonitor = ['0x123...', '0x456...', '0x789...'];

// Create a contract instance
const depositContract = new web3.eth.Contract([], depositContractAddress);

// Set up an event listener for the Deposit event
depositContract.events[depositEventName]({
  filter: { _to: addressesToMonitor }, // Specify the addresses to filter for
})
  .on('data', async (event) => {
    console.log('Deposit event:', event);
    
    // Handle the deposit event, e.g., trigger additional actions
  })
  .on('error', (error) => {
    console.error('Error:', error);
  });

This was probably the worst idea suggested by chatGPT. On my specific case were I have one address per user.

The first problem is that the addresses to monitor change by +1 every time a user register. So I would need to re-create the listener every new user registration. Another problem that I see is that the address list would eventually grow to tousands of elements (hopefully). But the worst problem is that there is no way to know that we lost elements on our listener. One server reset, network latency, anything could potencially make us lose deposit events, and we have no way ok knowing it since we don't track down the transaction count of each address.

I don't really see much use of this method for a payment system or gateway.

Inspect all block's transactions

To me this one made more sense, I would have tousands (hopefully) of addresses to track and I would like to be multitoken at some point, so the only way is to literally inspect every single transaction on the blockchain on realtime.

Some would say "but why don't you run your own node?" I don't want to. Its currently very hard since Shibarium source code is still private and most importatly is not necessary. RPC endpoints are open and free. So lets see how to do this.

Warning: lots of code ahead, lets try to get a rough idea first.

for ever
   get the last Shibarium block 
   for each transaction in block 
        if the transaction recipient address belongs to one of our users and has the contract address of the Shiba INU Token
            add the amount to the user's balance

There are some things I left outside the pseudocode, like:

  • ensure that we process blocks in order, starting from the last registered block in our database
  • beign able to turn on & off the daemon
  • schedule pulling blocks to avoid unecessary calls
  • handle errors

This is the actual code used in "production"

import assert from 'node:assert'
import Web3 from 'web3'
// import { isETHStrAddress } from './utils.js'
import * as queries from '../queries.js'
import logger from '../../../../lib/logger.js'
import { wallet } from '../../../../config.js'

// TODO: wait for x confirmations
// TODO: Handle Ethereum reorgs

const log = logger.child({ source: 'depositor/shibarium/daemon' })

const web3 = new Web3('https://www.shibrpc.com')

// Contract address of SHIBA INU Token
const shibContractAddress = '0x495eea66B0f8b636D441dC6a98d8F5C3D455C4c0'
  .toLowerCase()

const functionSignature = 'transfer(address,uint256)'

// if a transaction starts with this function selector its a ERC-20 transfer() call
// 0xa9059cbb
const functionSelector = web3.utils.keccak256(functionSignature).substring(
  0,
  10,
)

const LOG_EACH_BLOCKS = 8000

const AVERAGE_BLOCK_TIME = 5000

const PULL_INTERVAL = AVERAGE_BLOCK_TIME

// when error next pull internal is in x milliseconds
const ERROR_INTERVAL = 5000

let daemonOn = wallet.depositor.on

let timeoutId = null

let errorCount = 0

export const toggleDaemon = () => {
  daemonOn = !daemonOn

  if (daemonOn) {
    startBlockLoop()
  } else {
    clearTimeout(timeoutId)
  }
}

// Initialize with last block in db
let lastBlockCount = -1

if (daemonOn) {
  log.info({
    msg: 'starting Shibarium daemon',
    AVERAGE_BLOCK_TIME,
    ERROR_INTERVAL,
  })
  startBlockLoop()
}

async function startBlockLoop() {
  log.debug({ msg: 'initializing blook loop with last db block' })

  try {
    const lastBlock = await queries.getLastBlock()
    log.debug({ msg: 'lastBlock fetched db', lastBlock })
    lastBlockCount = lastBlock.height

    blockLoop()
  } catch (err) {
    log.error({ err, notice: 'failed to start block loop' })
    // crash the server on fail
    throw err
  }
}

// lapse = pull | error | immediate
function scheduleBlockLoop(lapse) {
  if (!daemonOn) {
    log.debug({ msg: 'stopping block loop, next block loop not scheduled' })
    return
  }

  const times = {
    pull: PULL_INTERVAL,
    error: ERROR_INTERVAL,
    immediate: 1,
  }
  const time = times[lapse]
  assert(time)
  log.debug({ msg: `scheduling blook loop in ${time}` })

  // TEMPORARY
  if (lapse === ERROR_INTERVAL) {
    log.info({ msg: 'retry disabled, stopping daemon' })
    return
  }

  log.debug({ msg: `next pull scheduled in ${time}` })
  setTimeout(blockLoop, time)
}

async function blockLoop() {
  const logf = log.child({ fn: 'blockLoop' })
  // let depositsFound

  try {
    // try fetching next block
    let nextBlock = undefined
    let nextBlockCount = lastBlockCount + 1
    try {
      nextBlock = await web3.eth.getBlock(nextBlockCount, true)
    } catch (err) {
      scheduleBlockLoop('error')
      return
    }

    if (!nextBlock) {
      // logf.debug({
      //   msg: 'no new block, scheduling pull for later',
      //   lastBlockCount,
      // })
      scheduleBlockLoop('pull')
      return
    }

    // logf.debug({
    //   msg: 'fetched new block',
    //   nextBlockCount,
    //   nextBlockHash: nextBlock?.hash,
    // })

    await processBlock(nextBlock)

    await queries.insertBlock(nextBlockCount)
    lastBlockCount = nextBlockCount

    if (lastBlockCount % LOG_EACH_BLOCKS === 0) {
      logf.notif({ msg: `Block ${lastBlockCount} processed sucessfully` })
    }
  } catch (err) {
    // TODO: check this comments
    // if the error is while processing txs some could be process and some could not be
    // we just reprocess the block again, anyway is recommended to manually check
    // block transactions in db
    //
    // if the error is while inserting block in db, we need to manually introduce it
    //
    // TODO: Detect network timeouts and network errors

    errorCount++

    // calls sometimes fail due to network issues
    log.error({
      err,
      notice:
        `failed to process block, scheduling next try in ${ERROR_INTERVAL}`,
      lastBlockCount,
      errorCount,
    })

    scheduleBlockLoop('error')

    return
  }

  // all good, process next block immediately
  // log.info({
  //   msg: 'block processed sucessfully',
  //   depositsFound,
  //   blockHeight: lastBlockCount,
  // })
  scheduleBlockLoop('immediate')
}

/**
 * @param {Object} block - hydrated block
 * if there's any error just throw and block will be re-processed
 *
 * TODO: if this functions fails we need to take urgent action
 * is there any way to make it more important?
 * for the time being it should be an error and we treat any error as bug
 */
async function processBlock(block) {
  const logf = log.child({ fn: 'processBlock' })

  // logf.debug({ msg: 'processing block' })

  const startDate = Date.now()
  let depositsFoundAmount = 0

  if (!block.transactions?.length) {
    return
  }

  try {
    for (const shibTx of block.transactions) {
      // found ERC-20 transfer() call
      if (shibTx.input?.toLowerCase().startsWith(functionSelector)) {
        // TODO: do all transactions have a to?
        if (!shibTx.to) {
          logf.warn({ msg: 'found transaction without to', shibTx })
          continue
        }

        if (shibTx.to.toLowerCase() === shibContractAddress) {
          // decode the address and the amount from the contract using eth ABI since its mostly the same
          const decodedData = web3.eth.abi.decodeParameters([
            'address',
            'uint256',
          ], shibTx.input.slice(10))

          const recipientAddress = decodedData[0]
          const amountTransferredWei = decodedData[1]
          assert(typeof amountTransferredWei === 'bigint')

          const userAddress = await queries.getUserAddress(recipientAddress)

          // if address belong to one of our users :) create a deposit
          if (userAddress) {
            assert(userAddress.user_id)

            const amountInShib = web3.utils.fromWei(
              amountTransferredWei,
              'ether',
            )
            assert(typeof amountInShib === 'string')

            logf.notif({
              msg: 'recieved user deposit',
              userId: userAddress.user_id,
              amountInShib,
            })

            await queries.addDeposit(
              userAddress.user_id,
              shibTx.hash,
              amountTransferredWei,
            )

            depositsFoundAmount++
          }
        }
      }
    }
  } catch (err) {
    const seconds = (Date.now() - startDate) / 1000
    log.error({
      err,
      notice: 'error processing transaction ids',
      secondsSpent: seconds,
    })
    throw err
  }

  //const seconds = (new Date() - startDate) / 1000
  //log.debug({
  //  msg: `processed ${transactionsIds.length}`,
  //  secondsSpent: seconds,
  //})

  return depositsFoundAmount
}