diff --git a/docs/0. Intro.md b/docs/0. Intro.md new file mode 100644 index 0000000..4c39199 --- /dev/null +++ b/docs/0. Intro.md @@ -0,0 +1,9 @@ +--- +id: intro +slug: /intro +title: Intro +--- + +Sourcify ([sourcify.dev](https://sourcify.dev)) is a source-code verification service for Ethereum smart contracts supporting Solidity and Vyper. Sourcify is fully commited to Open-source, Open-data and Open-standards in smart-contract verification instead of siloed, propriety services. We foster these values across the ecosystem and work actively to push the status-quo in this direction. + +Different than other verification services, Sourcify leverages the [Solidity metadata](/docs/metadata/) and its integrity hash to find [exact matches](/docs/exact-match-vs-match/) of Solidity contracts (see [the playground](https://playground.sourcify.dev)). diff --git a/docs/95. requestChainSupport.md b/docs/1. Requesting Chain Support.md similarity index 99% rename from docs/95. requestChainSupport.md rename to docs/1. Requesting Chain Support.md index dc42e53..c5664b9 100644 --- a/docs/95. requestChainSupport.md +++ b/docs/1. Requesting Chain Support.md @@ -1,5 +1,7 @@ --- slug: /chain-support +title: Requesting Chain Support +sidebar_position: 5 --- # Requesting Chain Support @@ -85,14 +87,12 @@ If you still want to request an EVM chain(s) to be supported by Sourcify, please 4. Install and build the project with `npm install` and `npx lerna run build`. 1. Provide test contracts: - 1. Deploy test contracts: 1. Using the provided script: You can easily deploy both contracts with an account private key using the provided [`script`](https://github.com/argotorg/sourcify/blob/staging/services/server/test/chains/deployContracts.ts), that can be run with `ts-node`. Inside `services/server`, run `npx ts-node test/chains/deployContracts.ts --help` for more information. 1. If you can't or don't want to paste your private key, deploy [this contract in the Thirdweb dashboard](https://thirdweb.com/0xAA6042aa65eb93C6439cDaeBC27B3bd09c5DFe94/Storage) through the wallet of your choice. 2. If you are not able to deploy new test contracts above (e.g. because it costs on a mainnet) you can provide existing contracts. Create a folder `services/server/test/chains/sources/{chainId}` and store the contract source files and metadata of already deployed contracts there. 1. Add the test for the new chains in `services/server/test/chains/chain-tests.spec.ts` similar to other chains. If you deployed a test contract as described above, the contract and the metadata are the following: - - Storage.sol: `shared/1_Storage.sol` and `shared/1_Storage.metadata.json` 1. To test locally run `export NEW_CHAIN_ID=1,11155111 && npm run test-local:chains` inside `services/server`. Note that this requires `docker-compose`. diff --git a/docs/1. Verify on Sourcify/1. What is source code verification.md b/docs/1. Verify on Sourcify/1. What is source code verification.md new file mode 100644 index 0000000..527228f --- /dev/null +++ b/docs/1. Verify on Sourcify/1. What is source code verification.md @@ -0,0 +1,31 @@ +--- +id: what-is-source-code-verification +slug: /what-is-source-code-verification +title: What is source code verification? +--- + +**Transparency**, **verifiability**, and **trustlessness** are the core values of blockchains and Ethereum especially. We want the smart contracts we are interacting with to be open-source. However you can't be sure if the open-source code you see is actually the one that lives on chain. Somone can show you a benign code on GitHub etc. and convince you to send your assets to a contract, but in reality it could be a malicious contract that's actually deployed at this address. + +This is where source code verification comes in. Source code verification makes sure the human-readable source code you see is the same as the one that was deployed on chain. + +:::info +A verified contract does not necessarily mean it is safe to interact with it. The verification does not look into _what_ the contract does, but only that it corresponds to this source-code. The source-code itself can be malicious and contain bugs. It is the auditors' and the community's responsibility to verify the code's security. +::: + +Smart contracts are written in human-readable programming languages like [Vyper](https://vyperlang.org/) or [Solidity](https://solidity.org/). But they are compiled to and deployed in bytecode (`1`s and `0`s), so they are not human-readable. + +``` +0x608060405234801561001057600080fd5b50600436106100365760003560e01c80632e64cec11461003b5780636057361d14610059575b600080fd5b610043610075565b60405161005091906100d9565b60405180910390f35b610073600480360381019061006e919061009d565b61007e565b005b60008054905090565b8060008190555050565b60008135905061009781610103565b92915050565b6000602082840312156100b3576100b26100fe565b5b60006100c184828501610088565b91505092915050565b6100d3816100f4565b82525050565b60006020820190506100ee60008301846100ca565b92915050565b6000819050919050565b600080fd5b61010c816100f4565b811461011757600080fd5b5056fea2646970667358221220404e37f487a89a932dca5e77faaf6ca2de3b991f93d230604b1b8daaef64766264736f6c63430008070033 +``` + +
+ How a contract looks like on Ethereum - 0x7ecedB5ca848e695ee8aB33cce9Ad1E1fe7865F8 on Ethereum Holesky Testnet +
+ +We can go from Solidity/Vyper code to bytecode, but not the other way around. This information is lost during compilation and we need the original source-code that compiles down to this bytecode. + +In simple terms, source code verification works by: + +1. Taking a smart-contract written in a human-readable programming language (Solidity/Vyper) +2. Compiling it down to bytecode +3. Comparing the compiled bytecode with the on-chain bytecode that is deployed at a certain chain and address. diff --git a/docs/1. Verify on Sourcify/2. Verify via UI.mdx b/docs/1. Verify on Sourcify/2. Verify via UI.mdx new file mode 100644 index 0000000..1f0a42e --- /dev/null +++ b/docs/1. Verify on Sourcify/2. Verify via UI.mdx @@ -0,0 +1,179 @@ +--- +id: how-to-verify +slug: /how-to-verify +title: Verify via UI +--- + +import TOCInline from "@theme/TOCInline"; +import chainAddress from "./chain-address.png"; +import importSources from "./import-sources.png"; +import verificationMethods from "./methods.png"; +import buildInfo from "./build-info.png"; +import jobError from "./job-error.png"; +import bytecodeDiff from "./bytecode-diff.png"; + +# Verify via UI + +You can verify using our new verification UI at https://verify.sourcify.dev. See the Github repo [here](https://github.com/sourcifyeth/verify.sourcify.dev). + +First choose the chain you want to verify the contract on and provide the contract address. + +
+ Chain and address +
+ +This will automatically check if the contract is verified on Sourcify on the same chain and all other supported chains. + +### Importing a contract + +You can import a verified contract from different sources: + +- **Etherscan**: Enter the chain and contract address to import the contract. You need an Etherscan API key to use this feature. Open the "Settings" on top right to add your API key. API keys are used locally in the browser and are not sent to Sourcify servers. +
+ Import sources +
+ +### Verification Methods + +For both Solidity and Vyper contracts you can choose between the following verification methods: + +- **Single file**: Provide single .sol or .vy file, alongside compiler settings. +- **Multiple files**: Provide multiple .sol or .vy files, alongside compiler settings. +- (preferred) **Standard JSON**: Provide a [standard JSON input file](https://docs.soliditylang.org/en/v0.8.30/using-the-compiler.html#input-description). + +Prefer using the standard JSON or framework commands directly (see below) to maintain the full compilation and avoid any changes. Different compiler settings can break the verification. Flattening the sources also breaks the metadata hashes and you won't be able to get an ["Exact Match"](/docs/exact-match-vs-match). + +Additionally the following option is available for Solidity contracts: + +- **Metadata.json**: Provide a [metadata.json file](/docs/metadata) to verify the contract in the first file input field. Provide all the source files in the second file input field. This option checks the source hashes in the metadata.json provided and compares them with the hashes of the source files provided before submitting the verification. + +
+ Verification methods +
+ Available verification methods +
+
+ +### Contract Identifier + +From the files and contracts provided, we will ask you to select the contract you want to verify. Ie. if you've written a contract called `MyContract.sol` using other contracts `Ownable.sol` and `ERC20.sol`, you will be asked to select `MyContract.sol` as the contract to verify. + +The format of the contract identifier is path:contractName e.g. + +``` +path/to/contractName.sol:contractName +``` + +The verification UI will parse the files and give you a list of contracts to choose from. You can always input the contract identifier manually. + +### (Optional) Creation Transaction Hash + +Sourcify needs to find the transaction the contract was deployed in to get the creation bytecode. We get this information from different sources like block explorers (see [here](https://github.com/argotorg/sourcify/blob/staging/services/server/src/sourcify-chains-default.json)). You can still provide a creation transaction hash manually if you want to, or if Sourcify fails to verify via creation bytecode. + +### Hardhat/Foundry build-info files + +If you’re using Hardhat or Foundry, the best way to verify is to use the framework’s CLI directly (see the respective documentation page to learn how). However, it is also possible to use the Sourcify build-info files to verify. + +The build-info files are generated by the frameworks and contain the compilation information. This can be parsed and used by the verify.sourcify.dev UI to verify the contract. + +You can find the build-info files in: + +- Hardhat: `artifacts/build-info/.json` +- Foundry: `out/build-info/.json` + +
+ Build info +
+ Build info files toggle +
+
+ +Click on the toggle to "Upload build-info file" and select the build-info file. + +## Verification Job + +Once you've provided all the information, you can submit the verification. This will create a verification job. You can see the job in the UI and get the verification result. + +If the verification fails you can see the error message and why it failed. + +
+ Job error +
+ Example of a verification job that failed +
+
+ +You can see the diffs of the bytecodes if the compilation succeeds but the bytecodes did not match. + +
+ Bytecode diff +
+ Bytecode diffs +
+
diff --git a/docs/1. Verify on Sourcify/3. Verify via Remix.md b/docs/1. Verify on Sourcify/3. Verify via Remix.md new file mode 100644 index 0000000..192b7bb --- /dev/null +++ b/docs/1. Verify on Sourcify/3. Verify via Remix.md @@ -0,0 +1,21 @@ +--- +id: verify-via-remix +slug: /verify-via-remix +title: Verify via Remix +--- + +You can use the "unified verification plugin" written by the Sourcify team to verify on multiple verifiers on Remix. + +1. Click on the "Plugins" tab in bottom of the sidebar and search for "Contract Verification" +2. Enable the plugin "Contract Verification" +3. Click on the "Contract Verification" tab in the sidebar +4. Enter the contract address and select a chain +5. Select the contract to verify (you need to have compiled a contract on Remix first) +6. By default all verifiers are enabled. For Etherscan you need to add your API key in the "Settings" tab +7. You can see the verification result in the "Receipts" tab + +
+ +
diff --git a/docs/1. Verify on Sourcify/4. Verify via Hardhat.md b/docs/1. Verify on Sourcify/4. Verify via Hardhat.md new file mode 100644 index 0000000..4e3082a --- /dev/null +++ b/docs/1. Verify on Sourcify/4. Verify via Hardhat.md @@ -0,0 +1,27 @@ +--- +id: verify-via-hardhat +slug: /verify-via-hardhat +title: Verify via Hardhat +--- + +The Hardhat plugin [@nomicfoundation/hardhat-verify](https://hardhat.org/hardhat-runner/plugins/nomicfoundation-hardhat-verify#verifying-on-sourcify) has official Sourcify support. + +(This is still using the legacy Sourcify API) + +In order to enable Sourcify support, add this to `hardhat.config.js`: + +```js title="hardhat.config.js" +sourcify: { + enabled: true, + // Optional: specify a different Sourcify server + apiUrl: "https://sourcify.dev/server", + // Optional: specify a different Sourcify repository + browserUrl: "https://repo.sourcify.dev", +} +``` + +After deploying your contracts you can verify on Sourcify with: + +```bash title="Verify a contract" +npx hardhat verify --network mainnet DEPLOYED_CONTRACT_ADDRESS +``` diff --git a/docs/1. Verify on Sourcify/5. Verify via Foundry.md b/docs/1. Verify on Sourcify/5. Verify via Foundry.md new file mode 100644 index 0000000..27cb41f --- /dev/null +++ b/docs/1. Verify on Sourcify/5. Verify via Foundry.md @@ -0,0 +1,33 @@ +--- +id: verify-via-foundry +slug: /verify-via-foundry +title: Verify via Foundry +--- + +Foundry natively supports Sourcify verification. You can verify contracts with the additional verify flags in Forge. + +(This is still using the legacy Sourcify API) + +```bash title="Deploy and verify a contract" +forge create --rpc-url \ + --constructor-args "ForgeUSD" "FUSD" 18 1000000000000000000000 \ + --private-key \ + src/MyToken.sol:MyToken \ + --verify \ + --verification-provider sourcify + --verifier-url https://localhost:5555 # optional, defaults to https://sourcify.dev/server/ +``` + +```bash title="Verify an already deployed (and compiled) contract" +forge verify-contract 0x55f7d4279CE387067f12561e7E0c194f5186cFba \ + src/MyToken.sol:MyToken \ + --chain-id 11155111 \ + --verifier sourcify + --verifier-url https://localhost:5555 # optional, defaults to https://sourcify.dev/server/ +``` + +```bash title="Check if a contract is verified" +forge verify-check 0x55f7d4279CE387067f12561e7E0c194f5186cFba \ + --chain-id 11155111 \ + --verifier sourcify +``` diff --git a/docs/1. Verify on Sourcify/6. Verify via API.md b/docs/1. Verify on Sourcify/6. Verify via API.md new file mode 100644 index 0000000..6fdcc60 --- /dev/null +++ b/docs/1. Verify on Sourcify/6. Verify via API.md @@ -0,0 +1,7 @@ +--- +id: verify-via-api +slug: /verify-via-api +title: Verify via API +--- + +See the [API docs](/docs/api) for more information. diff --git a/docs/96. supportedChains.mdx b/docs/1. Verify on Sourcify/7. Supported chains.md similarity index 96% rename from docs/96. supportedChains.mdx rename to docs/1. Verify on Sourcify/7. Supported chains.md index 834d03a..ea1ef54 100644 --- a/docs/96. supportedChains.mdx +++ b/docs/1. Verify on Sourcify/7. Supported chains.md @@ -1,6 +1,7 @@ --- id: chains slug: /chains +title: Supported chains --- import Table from "./supportedChainsTable.tsx"; @@ -18,4 +19,3 @@ _If the tests for a chain will fail two weeks in a row, we will assume the chain
- diff --git a/docs/1. Verify on Sourcify/8. Supported languages.md b/docs/1. Verify on Sourcify/8. Supported languages.md new file mode 100644 index 0000000..9e4fe51 --- /dev/null +++ b/docs/1. Verify on Sourcify/8. Supported languages.md @@ -0,0 +1,15 @@ +--- +id: supported-languages +slug: /supported-languages +title: Supported languages +--- + +# Supported languages + +Sourcify supports Solidity, Vyper, and Yul. The table below summarizes the current level of support per language. + +| Language | Supported | Exact match support | Monitoring | Metadata/ABI | IPFS data | +| --- | --- | --- | --- | --- | --- | +| Solidity | ✅ | ✅ | Similarity and IPFS | Both | ✅ | +| Vyper | ✅ | ❌ | Similarity | ABI | ❌ | +| Yul | ✅ | ❌ | Similarity | ❌ | ❌ | diff --git a/docs/1. Verify on Sourcify/_category_.json b/docs/1. Verify on Sourcify/_category_.json new file mode 100644 index 0000000..0d44758 --- /dev/null +++ b/docs/1. Verify on Sourcify/_category_.json @@ -0,0 +1,3 @@ +{ + "collapsed": false +} diff --git a/docs/3. Verification/build-info.png b/docs/1. Verify on Sourcify/build-info.png similarity index 100% rename from docs/3. Verification/build-info.png rename to docs/1. Verify on Sourcify/build-info.png diff --git a/docs/3. Verification/bytecode-diff.png b/docs/1. Verify on Sourcify/bytecode-diff.png similarity index 100% rename from docs/3. Verification/bytecode-diff.png rename to docs/1. Verify on Sourcify/bytecode-diff.png diff --git a/docs/3. Verification/chain-address.png b/docs/1. Verify on Sourcify/chain-address.png similarity index 100% rename from docs/3. Verification/chain-address.png rename to docs/1. Verify on Sourcify/chain-address.png diff --git a/docs/3. Verification/import-sources.png b/docs/1. Verify on Sourcify/import-sources.png similarity index 100% rename from docs/3. Verification/import-sources.png rename to docs/1. Verify on Sourcify/import-sources.png diff --git a/docs/3. Verification/job-error.png b/docs/1. Verify on Sourcify/job-error.png similarity index 100% rename from docs/3. Verification/job-error.png rename to docs/1. Verify on Sourcify/job-error.png diff --git a/docs/3. Verification/methods.png b/docs/1. Verify on Sourcify/methods.png similarity index 100% rename from docs/3. Verification/methods.png rename to docs/1. Verify on Sourcify/methods.png diff --git a/docs/1. Verify on Sourcify/supportedChainsTable.tsx b/docs/1. Verify on Sourcify/supportedChainsTable.tsx new file mode 100644 index 0000000..153b253 --- /dev/null +++ b/docs/1. Verify on Sourcify/supportedChainsTable.tsx @@ -0,0 +1,334 @@ +import React, { useEffect, useState, useMemo } from "react"; +import { renderToString } from "react-dom/server"; +import ReactTooltip from "react-tooltip"; +import LoadingOverlay from "../../src/components/LoadingOverlay"; +import Chart from "../Chart"; + +const Yes = () =>
✅
; +const No = () =>
❌
; +const C = ({ children }) =>
{children}
; +const R = ({ children }) =>
{children}
; + +const TestResult = ({ result, detailedUrl, type, error }) => { + const ToolTip = ( + + {`${type} contract verification for the chain failed`} +
+ See{" "} + + CI test run for details + + + ); + if (error) { + return ( +
+ ⚠️ +
+ ); + } + if (result === true) + return ( +
+ ✅ +
+ ); + if (result === false) + return ( + <> + +
+ ❌ +
+ + ); + if (result === undefined) + return ( +
+ 🤷 +
+ ); + return null; +}; + +const Table = () => { + const [sourcifyChains, setSourcifyChains] = useState(); + const [error, setError] = useState(""); + const [testReportObject, setTestReportObject] = useState(); + const [testMap, setTestMap] = useState(); + const [testDate, setTestDate] = useState(); + const [stats, setStats] = useState(); + + const sourcifyChainMap = useMemo(() => { + return sourcifyChains?.reduce(function (acc, currentChain) { + acc[currentChain.chainId] = currentChain; + return acc; + }, {}); + }, [sourcifyChains]); + + const addMonitoredSupportFrom = async (url, supportedChains) => { + try { + // Try to fetch monitor information + const req = await fetch(url); + const res = await req.json(); + res.forEach((monitoredChain) => { + const foundChain = supportedChains.find((supportedChain) => supportedChain.chainId === monitoredChain.chainId); + if (!foundChain) return; // Could be that a chain that is not supported by Sourcify + foundChain.monitored = true; + }); + } catch (e) { + setError("Cannot get monitor data from url: " + url); + } + }; + + useEffect(() => { + fetch("https://sourcify.dev/server/chains") + .then((res) => res.json()) + .then(async (chains) => { + await addMonitoredSupportFrom( + "https://raw.githubusercontent.com/argotorg/sourcify/staging/services/monitor/monitorChains.json", + chains + ); + sortChains(chains); + setSourcifyChains(chains); + }) + .catch((err) => setError("Error fetching chains from the Sourcify server\n\n" + err.message)); + fetch("https://sourcify.dev/server/chain-tests") + .then((response) => { + if (!response.ok) { + throw new Error("Error fetching chain tests \n\n"); + } + return response.json(); + }) + .then((json) => { + setTestDate(json.testReport.stats.end); + const testMap = formatRawTestReport(json.testReport); + setTestMap(testMap); + setTestReportObject(json); + }) + .catch((err) => setError(err.message)); + }, []); + + useEffect(() => { + fetch(`https://sourcify.dev/static/stats.json`) + .then((res) => res.json()) + .then((json) => setStats(json)) + .catch(() => {}); + }, []); + + function sortChains(chains) { + return chains.sort((a, b) => { + const ETHEREUM_CHAINS = [1, 5, 11155111, 1700, 3, 4]; + const hasEthereumA = ETHEREUM_CHAINS.includes(parseInt(a.chainId)); + const hasEthereumB = ETHEREUM_CHAINS.includes(parseInt(b.chainId)); + + const isMonitoredA = a.monitored === true; + const isMonitoredB = b.monitored === true; + + // Ethereum chains on top + if (hasEthereumA && !hasEthereumB) { + return -1; + } else if (!hasEthereumA && hasEthereumB) { + return 1; + } + + // 'monitored' come next + if (isMonitoredA && !isMonitoredB) { + return -1; + } else if (!isMonitoredA && isMonitoredB) { + return 1; + } + + // Chains with trace support come next + const hasTraceA = a?.traceSupportedRPCs?.length > 0; + const hasTraceB = b?.traceSupportedRPCs?.length > 0; + + if (hasTraceA && !hasTraceB) { + return -1; + } else if (!hasTraceA && hasTraceB) { + return 1; + } + + // Chains with Etherscan API come next + const hasEtherscanA = a?.etherscanAPI; + const hasEtherscanB = b?.etherscanAPI; + + if (hasEtherscanA && !hasEtherscanB) { + return -1; + } else if (!hasEtherscanA && hasEtherscanB) { + return 1; + } + + // Sort the rest alphabetically by chain.name + return a.name.localeCompare(b.name); + }); + } + + // Takes the raw mochawesome test report .json and formats with the result of the standard and immutable contract verification for each chain. + const formatRawTestReport = (rawReport) => { + const testsArr = rawReport.results[0].suites[0].tests; + const testMap = {}; + testsArr.forEach((test) => { + const context = JSON.parse(test.context); + if (!context) return; + const chainId = context.value.chainId; + const testType = context.value.testType; // either "normal" or "immutable" + if (!testMap[chainId]) testMap[chainId] = {}; + testMap[chainId][testType] = test.pass; + }); + return testMap; + }; + + if (!sourcifyChains) { + return ( +
+ +
+ ); + } + if ((!testMap || !testReportObject) && !error) { + return ( +
+ +
+ ); + } + + const testRunCircleURL = + testReportObject && + `https://app.circleci.com/pipelines/github/argotorg/sourcify/${testReportObject.pipelineNumber}/workflows/${testReportObject.workflowId}/jobs/${testReportObject.jobNumber}`; + const testReportHtmlURL = + testReportObject && + `https://dl.circleci.com/private/output/job/${testReportObject.jobId}/artifacts/0/chain-tests-report/report.html`; + + const rows = sourcifyChains.map((chain, i) => { + return ( + + + + + + + + + ); + }); + return ( + <> + {testReportObject && ( +

+ You can check out the complete{" "} + + HTML report + {" "} + and the{" "} + + detailed CI output + {" "} + of the chain tests. Tested on: {testDate} +

+ )} + +
+ {stats ? ( + <> +

Chains by Verified Contracts

+ + + ) : ( +
+ +
+ )} +
+
+

Chains by Type of Support

+ {error &&
{error}
} + {sourcifyChains.length > 0 && ( +
+ {" "} + Current number of EVM chains and their support type: +
    +
  • + Monitoring (i.e. automatic verification) and Verification support:{" "} + {sourcifyChains.filter((c) => c.monitored).length} +
    • +
  • + Verification support: {sourcifyChains.filter((c) => c.supported).length} +
    • +
  • + That has an RPC with trace support:{" "} + {sourcifyChains.filter((c) => c.traceSupportedRPCs?.length > 0).length} +
    • +
  • + Not Supported (deprecated): {sourcifyChains.filter((c) => !c.supported).length} +
    • +
  • + Total: {sourcifyChains.length} +
    • +
+
+ )} + +
{chain.title || chain.name} + {chain.chainId} + + {chain.supported ? "Verification" : "Not Supported"}
+ {chain.monitored ? "Monitoring" : ""} + 		{chain?.traceSupportedRPCs?.length > 0 ? "✅" : ""}{chain.etherscanAPI ? "✅" : ""} + { + + } +
+ + + + + + + + + + + {rows} +
ChainChain IDSupport Type +

This allows verifying contracts created with factories with the creation bytecode.

+

+ If not supported, contracts can be verified normally with the runtime bytecode, but also with the + creation bytecode if they were created by an EOA, ie. the creation bytecode is in the transaction + payload that created the contract. +

+ + )} + data-for="rpc-trace-support" + > + RPCs with trace support{" "} + + ? + + 		Import from EtherscanVerification Tests
+
+ + ); +}; + +export default Table; diff --git a/docs/1. intro.mdx b/docs/1. intro.mdx deleted file mode 100644 index f65b971..0000000 --- a/docs/1. intro.mdx +++ /dev/null @@ -1,8 +0,0 @@ ---- -description: Sourcify Documentation and Blog Page -title: Sourcify Docs ---- - -import EmbedReadmeIntro from "./EmbedReadmeIntro"; - - \ No newline at end of file diff --git a/docs/100. Additional Resources.md b/docs/100. Additional Resources.md deleted file mode 100644 index eb3b63d..0000000 --- a/docs/100. Additional Resources.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -id: additional-resources -title: Additional Resources -slug: /additional-resources ---- - -# Additional Resources - -Here are related projects, efforts, and discussions - -## Human-readable contract interactions & signatures - -- [EIP-712: Ethereum typed structured data hashing and signing](https://eips.ethereum.org/EIPS/eip-712) -- [Trustless Signing UI Protocol](https://github.com/ethereum/EIPs/issues/719) -- [EIP-4430: Described Transactions](https://eips.ethereum.org/EIPS/eip-4430) - - A technique for contracts to provide a human-readable description of a transaction's side-effects. - - Discussion (closed): https://github.com/ethereum/EIPs/issues/4431 - - Discussion: https://ethereum-magicians.org/t/discussion-eip-4430-described-transactions/8762 -- [EIP-3224: Described Data](https://eips.ethereum.org/EIPS/eip-3224) - - Spun out of EIP-4430. Contract method to compute human-readable descriptions for signable data - - Discussion: https://github.com/ethereum/EIPs/issues/3225 -- Metamask [Transaction Insights](https://metamask.zendesk.com/hc/en-us/articles/4412543412123-Transaction-Insight) -- [(Draft) EIP: Rich Site-Proposed Contract Metadata](https://ethereum-magicians.org/t/eip-rich-site-proposed-contract-metadata/9635) - -## Source Code Verification - -- [WorkShop Notes: ChainAgnostic Verification Standard](https://github.com/ChainAgnostic/AMS-CASA-gathering/issues/20) diff --git a/docs/3. Verification/2. exactMatchVsMatch.md b/docs/2. Sourcify in depth/1. Exact match vs match.md similarity index 100% rename from docs/3. Verification/2. exactMatchVsMatch.md rename to docs/2. Sourcify in depth/1. Exact match vs match.md diff --git a/docs/9. metadata.mdx b/docs/2. Sourcify in depth/2. Solidity metadata.md similarity index 89% rename from docs/9. metadata.mdx rename to docs/2. Sourcify in depth/2. Solidity metadata.md index d78412e..281b4ab 100644 --- a/docs/9. metadata.mdx +++ b/docs/2. Sourcify in depth/2. Solidity metadata.md @@ -1,18 +1,18 @@ --- id: metadata -title: Contract Metadata +title: Solidity metadata slug: /metadata --- -import AvailableOnlyForSolidityAdmonition from "./AvailableOnlyForSolidityAdmonition"; +import AvailableOnlyForSolidityAdmonition from "../AvailableOnlyForSolidityAdmonition"; -# Contract metadata +# Solidity contract metadata ## Metadata -Since [v0.4.7](https://github.com/ethereum/solidity/releases/tag/v0.4.7) (2016-12-15) Solidity compiler generates an output file called metadata that contains information about the contract and the compilation ([see Solidity docs](https://docs.soliditylang.org/en/latest/metadata.html)). This includes the compiler settings, [contract ABI](https://docs.soliditylang.org/en/latest/abi-spec.html), [NatSpec contract documentation](https://docs.soliditylang.org/en/latest/natspec-format.html), and used source files and their hashes. Sourcify takes full advantage of this information to [exactly verify](/docs/exact-match-vs-match) a deployed contract i.e. a byte-by-byte match of the contract. +Since [v0.4.7](https://github.com/ethereum/solidity/releases/tag/v0.4.7) (2016-12-15) Solidity compiler generates an output file called metadata that contains information about the contract and the compilation ([see Solidity docs](https://docs.soliditylang.org/en/latest/metadata.html)). This includes the compiler settings, [contract ABI](https://docs.soliditylang.org/en/latest/abi-spec.html), [NatSpec contract documentation](https://docs.soliditylang.org/en/latest/natspec-format.html), and used source files and their hashes. Sourcify takes full advantage of this information to [exactly verify](/docs/exact-match-vs-match) a deployed contract i.e. a byte-by-byte match of the contract. To see metadata in action check out the [Metadata Playground](https://playground.sourcify.dev). @@ -51,22 +51,22 @@ To have the metadata in the ouput of the compilation with Solidity's [standard J ... } ``` + ### Foundry Foundry stores the contract compilation outputs to `out/CONTRACT_NAME` folder. The `.json` file contains the metadata of the contract under `"rawMetadata"` and `"metadata"` fields. However you don't need to extract the metadata manually for verification. -You can use the `forge verify-contract` command to verify a contract. See [Foundry verification](/docs/how-to-verify/#foundry) for more details. +You can use the `forge verify-contract` command to verify a contract. See [Foundry verification](/docs/verify-via-foundry) for more details. Simply drag and drop the output `.json` file of the contract you want to verify to the [sourcify.dev](https://sourcify.dev) UI or provide it as the metadata file to the [API](/docs/api). - ### Hardhat -You can use the native Hardhat [verify plugin](https://hardhat.org/hardhat-runner/plugins/nomicfoundation-hardhat-verify#verifying-on-sourcify) to verify your contracts. See [Hardhat verification](/docs/how-to-verify/#hardhat-verify-plugin) for more details. +You can use the native Hardhat [verify plugin](https://hardhat.org/hardhat-runner/plugins/nomicfoundation-hardhat-verify#verifying-on-sourcify) to verify your contracts. See [Hardhat verification](/docs/verify-via-hardhat) for more details. Hardhat stores the outputs of the compilations under the `artifacts/build-info/` folder inside the project. The `.json` file under the folder contains the [Standard JSON Input-Output](https://docs.soliditylang.org/en/latest/using-the-compiler.html#compiler-input-and-output-json-description) of the Solidity compiler for all contracts. -Since [v2.6.8](https://github.com/nomiclabs/hardhat/releases/tag/hardhat%402.6.8), Hardhat outputs the metadata files of each contract by default under the JSON's `.output.contracts...metadata` field. You don't have to extract the metadata manually. To verify via the Hardhat output you can provide the JSON containing the artifacts to Sourcify via UI or API. See [Hardhat verification](/docs/how-to-verify/#with-hardhat-output) for more details. If the file is too large, try zippping it and uploading the zip file. +Since [v2.6.8](https://github.com/nomiclabs/hardhat/releases/tag/hardhat%402.6.8), Hardhat outputs the metadata files of each contract by default under the JSON's `.output.contracts...metadata` field. You don't have to extract the metadata manually. To verify via the Hardhat output you can provide the JSON containing the artifacts to Sourcify via UI or API. See [Hardhat verification](/docs/verify-via-hardhat) for more details. If the file is too large, try zippping it and uploading the zip file. ### Hardhat - earlier than v2.6.8 diff --git a/docs/4. repository/1. sourcify-database.mdx b/docs/2. Sourcify in depth/3. Database.md similarity index 90% rename from docs/4. repository/1. sourcify-database.mdx rename to docs/2. Sourcify in depth/3. Database.md index 6fa39e8..7a08206 100644 --- a/docs/4. repository/1. sourcify-database.mdx +++ b/docs/2. Sourcify in depth/3. Database.md @@ -1,3 +1,9 @@ +--- +id: sourcify-database +slug: /repository/sourcify-database/ +title: Database +--- + # Sourcify Database Sourcify Database is the main storage backend for Sourcify. It is a PostgreSQL database that follows the [Verifier Alliance Schema](https://github.com/verifier-alliance/database-specs) as its base with few modifications. @@ -66,11 +72,3 @@ Other known inconsistencies in the data below (not planned to fix) are documente ## Download See [Download the Dataset](/docs/repository/download-dataset/) for instructions on downloading the database in Parquet format. - -## BigQuery Dataset - -We also provide a public BigQuery dataset for convenient querying and exploration: - -[Sourcify BigQuery dataset](https://console.cloud.google.com/bigquery/analytics-hub/exchanges/projects/1019539084286/locations/europe-west1/dataExchanges/sourcify_19a0c79ef3a/listings/sourcify_19a0c7d0be2?project=tranquil-petal-125711) - -The dataset is updated continuously as new contracts are verified. You need a Google account to access it. diff --git a/docs/3. Verification/3. libraries.md b/docs/2. Sourcify in depth/4. Transformations/1. libraries.md similarity index 90% rename from docs/3. Verification/3. libraries.md rename to docs/2. Sourcify in depth/4. Transformations/1. libraries.md index 80d0443..dca5ab3 100644 --- a/docs/3. Verification/3. libraries.md +++ b/docs/2. Sourcify in depth/4. Transformations/1. libraries.md @@ -4,7 +4,7 @@ title: Libraries slug: /libraries --- -import AvailableOnlyForSolidityAdmonition from "../AvailableOnlyForSolidityAdmonition"; +import AvailableOnlyForSolidityAdmonition from "../../AvailableOnlyForSolidityAdmonition"; # Verification with Libraries @@ -26,7 +26,7 @@ Normally when a contract has linked libraries these are noted in the `libraries` These contract addresses for linked libraries need to be [passed to the compiler for linking](https://docs.soliditylang.org/en/latest/using-the-compiler.html#library-linking) prior to the compilation, and should not be replaced directly in the bytecode. [See the Warning in Solidity docs](https://docs.soliditylang.org/en/latest/using-the-compiler.html#library-linking). -However some tools and frameworks do regex replace the placeholders in the bytecode. This causes the addresses not to be found in the metadata. In that case, Sourcify compares the onchain deployed bytecode with the compiled deployed bytecode (with placeholders) to extract the linked library addresses. This will be saved in a file called `library-map.json` in the repository. +However some tools and frameworks do regex replace the placeholders in the bytecode. This causes the addresses not to be found in the metadata. In that case, Sourcify compares the onchain deployed bytecode with the compiled deployed bytecode (with placeholders) to extract the linked library addresses. Check out these examples: diff --git a/docs/3. Verification/4. immutables.md b/docs/2. Sourcify in depth/4. Transformations/2. Immutables.md similarity index 98% rename from docs/3. Verification/4. immutables.md rename to docs/2. Sourcify in depth/4. Transformations/2. Immutables.md index 72082dd..d0569e7 100644 --- a/docs/3. Verification/4. immutables.md +++ b/docs/2. Sourcify in depth/4. Transformations/2. Immutables.md @@ -38,6 +38,7 @@ For example in the bytecode excerpt from the Görli contract 0xbdde4d595f2cdda92 Sourcify supports verification of Vyper contracts containing immutable variables. Here's how it works: 1. **CBOR auxdata**: Vyper appends an array in CBOR format at the end of the creation bytecode. This array contains the following elements: + ``` [ integrity_hash, // Hash of the contract's source code, included starting with Vyper 0.4.1 @@ -56,4 +57,4 @@ Sourcify supports verification of Vyper contracts containing immutable variables 3. **Verification Process**: To verify the contract: - Remove the immutable values from the onchain runtime bytecode (by truncating the last `immutable_size` bytes) - - Compare this with the recompiled bytecode \ No newline at end of file + - Compare this with the recompiled bytecode diff --git a/docs/2. Sourcify in depth/4. Transformations/EmbedReadmeTransformations.jsx b/docs/2. Sourcify in depth/4. Transformations/EmbedReadmeTransformations.jsx new file mode 100644 index 0000000..5919dda --- /dev/null +++ b/docs/2. Sourcify in depth/4. Transformations/EmbedReadmeTransformations.jsx @@ -0,0 +1,19 @@ +import EmbedReadme from "../../EmbedReadme"; +import React from "react"; +const repo = "verifier-alliance/verifier-alliance.github.io"; +const branch = "main"; +const readmePath = "/docs/3. transformations.md"; +const rawBaseUrl = "https://raw.githubusercontent.com"; +const pageBaseUrl = "https://github.com"; + +export default function() { + return ( + + ); +} diff --git a/docs/2. Sourcify in depth/4. Transformations/index.md b/docs/2. Sourcify in depth/4. Transformations/index.md new file mode 100644 index 0000000..530ee44 --- /dev/null +++ b/docs/2. Sourcify in depth/4. Transformations/index.md @@ -0,0 +1,9 @@ +--- +id: transformations +slug: /transformations +title: Transformations +--- + +import EmbedReadmeTransformations from "./EmbedReadmeTransformations"; + + diff --git a/docs/3. Verification/10.similarityVerification.md b/docs/2. Sourcify in depth/5. Similarity verification.md similarity index 100% rename from docs/3. Verification/10.similarityVerification.md rename to docs/2. Sourcify in depth/5. Similarity verification.md diff --git a/docs/3. Verification/11. externalVerifiers.md b/docs/2. Sourcify in depth/6. Automatic verification on external verifiers.md similarity index 98% rename from docs/3. Verification/11. externalVerifiers.md rename to docs/2. Sourcify in depth/6. Automatic verification on external verifiers.md index db418d1..d9d9cfe 100644 --- a/docs/3. Verification/11. externalVerifiers.md +++ b/docs/2. Sourcify in depth/6. Automatic verification on external verifiers.md @@ -58,5 +58,5 @@ Remember also to specify the API keys. BLOCKSCOUT_API_KEY= ROUTESCAN_API_KEY= # warn: Etherscan have a very strict rate limit of 200 verifications / day. -ETHERSCAN_API_KEY= -``` \ No newline at end of file +ETHERSCAN_API_KEY= +``` diff --git a/docs/6. monitoring.md b/docs/2. Sourcify in depth/7. Monitoring service.md similarity index 96% rename from docs/6. monitoring.md rename to docs/2. Sourcify in depth/7. Monitoring service.md index dd699ec..c39ccda 100644 --- a/docs/6. monitoring.md +++ b/docs/2. Sourcify in depth/7. Monitoring service.md @@ -4,7 +4,7 @@ title: Monitoring Service slug: /monitoring --- -import AvailableOnlyForSolidityAdmonition from "./AvailableOnlyForSolidityAdmonition"; +import AvailableOnlyForSolidityAdmonition from "../AvailableOnlyForSolidityAdmonition"; # Monitoring Service diff --git a/docs/20. databaseMigration.md b/docs/20. databaseMigration.md deleted file mode 100644 index ce6eb3a..0000000 --- a/docs/20. databaseMigration.md +++ /dev/null @@ -1,237 +0,0 @@ ---- -id: database-migration -title: Migration from Filesystem to Database -slug: /database-migration ---- - -# Migration from Filesystem to Database - -:::info -This page describes the process of migrating all the contracts from the filesystem based repository to a new storage structure. -It is meant for anyone running their own Sourcify instance. -::: - -## Current repository limitations - -Currently Sourcify stores verified contracts into a repository: - -- in which there are contracts verified with old verification and storage strategies: [#902](https://github.com/argotorg/sourcify/issues/902), [#1071](https://github.com/argotorg/sourcify/issues/1071) -- in which the sources are stored mirroring the original file system, causing naming path problems: [#515](https://github.com/argotorg/sourcify/issues/515) -- filesystem based, causing a difficult maintenance and expansive advanced searches (e.g. by contract's name) - -## New storage structure - -We cannot move completely to a database dropping the filesystem because we want to keep serving metadata files and sources over IPFS. So the new storage structure includes: - -- **RepositoryV1 (deprecated)**: legacy filesystem-based repository for verified contracts. We kept this around as the old [repo.sourcify.dev](https://repo.sourcify.dev) was based on [h5ai](https://github.com/sourcifyeth/h5ai-nginx) a filesystem-based web server. -- **RepositoryV2**: a filesystem-based repository where sources for each contract are stored in its `sources/` folder, without replicating the contract's original filesystem structure. Instead, it names files using `keccak256(file: string)`. This serves the purpose of serving the sources over IPFS. -- **Database**: a PostgreSQL database based on [the schema of the Alliance Database](https://github.com/verifier-alliance/database-specs) customized to better fit Sourcify (See [Sourcify Database](/docs/repository/sourcify-database)). - -## Migration process - -Sourcify will keep storing and serving matches from the old repository until the migration is completed. The migration process is structured as follows: - -1. **Enable RepositoryV2 and Database in `sourcify-server`**, `sourcify-server` will store the new matches into RepositoryV2 and the Database (including the table `sourcify_sync`). `sourcify-server` will continue writing into _current_repositoryV1_. -2. **Import all the contracts** from _current_repositoryV1_ into `sourcify_sync` -3. **Start the migration script**, the script reads all `synced=false` contracts from `sourcify_sync` and sends for each of them a verification request to the dedicated `sourcify-server-migration` instance that writes to: - 1. the same RepositoryV2 as `sourcify-server` - 2. the same Database as `sourcify-server` - 3. the _new_repositoryV1_ -4. **Start using the new_repositoryV1:** After the migration is completed, on the dedicated `sourcify-server-migration` instance, the new version of repositoryV1 will be produced (_new_repositoryV1_). Before using _new_repositoryV1_ in `sourcify-server`, all the contracts added to the _current_repositoryV1_ after starting the migration script need to be moved to _new_repositoryV1_. - -:::info sourcify-server-migration -A temporary new Sourcify’s server is used in the migration process to not overload the main server with all the previous requests, we call this new server `sourcify-server-migration`. -::: - -:::info sourcify_sync -The migration process is tracked and managed by using a table named `sourcify_sync` that contains: - -- `chain_id` -- `address` -- `created_at` -- `synced`: the status of the contract's migration, _"has the contract been migrated to the new storage structure?"_ - -::: - -:::info current_repositoryV1 vs new_repositoryV1 -As explained above, after we complete the migration, RepositoryV1 will be a refined version of the current repository containing contracts stored with new verification and storage strategies. -In the process of the migration: - -- `sourcify-server` will keep writing RepositoryV1’s contracts to the current repository, in this document we will refer to the current repository as _current_repositoryV1_. -- `sourcify-server-migration` will store RepositoryV1’s contracts to a new empty repository, in this document we will refer to this new repository as _new_repositoryV1_. - -::: - -### 1. Enabling RepositoryV2 and Database - -- **a. Start PostgreSQL** -- **b. Migrate the database schema** - - We first need to write the existing schemas defined under `services/database/migrations` into our fresh new Postgres DB: - - - I. Clone the Sourcify monorepo and run `npm install` - - II. `cd services/database` - - III. Edit `database.json` to configure the connection to the PostgreSQL database - - IV. `npm run migrate:up -- --env=YOUR_ENV`: Instead of `YOUR_ENV`, put the JSON key under which you defined the database connection in `database.json` in the previous step. - -- **c. Enable RepositoryV2** - - Make sure that in your configuration file inside `services/server/src/config` there is the `repositoryV2` configuration. Set the path accordingly: - - ```json - repositoryV2: { - path: "/home/app/repositoryV2", - } - ``` - -- **d. Enable Database** - - Make sure to provide the environment variables to enable the database. Set the variables accordingly in sourcify-server’s `.env` file: - - ``` - SOURCIFY_POSTGRES_HOST=postgresql - SOURCIFY_POSTGRES_DB=sourcify - SOURCIFY_POSTGRES_USER=sourcify - SOURCIFY_POSTGRES_PORT=5432 - SOURCIFY_POSTGRES_PASSWORD=sourcify - ``` - -- **e. Check that everything is working** - - Restart `sourcify-server` - - Make sure that for every match: - - There is a row into `sourcify_matches` , `verified_contracts` (and related tables), as well as `sourcify_sync` - - There is the respective folder in `repositoryV2/contracts/:match_type/:chain_id/:address` - -### 2. Importing existing contracts - -- **a. Importing contracts from the old repository** - - I. `cd services/database` - - II. Make sure to create a `.env` file to configure the connection to the PostgreSQL database. There is already a `.env.template` for reference. - - III. `npm run sourcify:database import-repo /data/repository/contracts/`: Instead of `/data/repository/contracts/`, put the current repository’s path -- **b. Wait** - - This process can take hours. It’s normal to not see any output after running the script for a while. - - Wait until the scripts exits with `successfully imported from “:repo_path” :number_imported_contracts contracts.` - -### 3. Start the migration instance and the migration script - -- **a. Start a dedicated `sourcify-server-migration` instance** - - I. ⚠ You need to checkout **6cdcdce8f186cdf3c90438393fee1efcbe272622** or `git checkout tags/sourcify-server@1.6.0` - - II. Configure it accordingly using the same steps described in **1. Enabling RepositoryV2 and Database**. Both `sourcify-server` and `sourcify-server-migration` should write to the same repoV2 and Database. - - III. ⚠ In the configuration of `sourcify-server-migration`, change the path of repositoryV1 to a new path: this is the _new_repositoryV1_. Don’t write in _current_repositoryV1_! - - IV. Start the new `sourcify-server-migration` instance -- **b. Find unsupported chains** - - I. From `services/server/src/sourcify-chains-default.json` or the `sourcify-chains` file that you are using, find all the chainIds of unsupported chains. - - II. Prepare them formatted in a list separated by a comma. E.g. `0,3,4,42,77,300,534,421613,4216137055,420,28,43,486217935,356256156,103090,420666,420420,99,592,8217,28528,14,421611,69,167005,167006` - - III. ⚠ Remember to include chainId 0 in this list -- **c. Start the synchronization script** - - I. `cd services/database` - - II. Edit `database.json` to configure the connection to the PostgreSQL database - - III. `npm run sourcify:database sync URL_TO_SOURCIFY_MIGRATION_INSTANCE /data/repository/contracts/ -- -c LIST_OF_UNSUPPORTED_CHAINS --limit 5` - - Instead of `/data/repository/contracts/`, put the current repository’s path. The script needs to access the old repo to get the file used to send the verification request. - - Instead of `LIST_OF_UNSUPPORTED_CHAINS`, put the formatted list prepared in the previous step. The script will not send verification requests to these chains. - - You can configure `--limit 5`: limit is the number of concurrent verifications that are going to be sent to the `sourcify-server-migration` for each chain. If you set 5, 5 requests are going to be sent for each chain. -- **d. Wait and monitor** - - The script will update migrated contracts into `sourcify_sync` setting `synced=true`. - - The script will take days and a small portion of old contracts will not be verified because of the new verification and storage strategies. - - If the script is interrupted for any reason, you can restart it. - - You can use this query to keep track of the current status: - ```sql - select - chain_id, - case (chain_id in (0,3,4,42,77,300,534,421613,4216137055,420,28,43,486217935,356256156,103090,420666,420420,99,592,8217,28528,14,421611,69,167005,167006)) when true then 'unsupported' else '' end, -- paste here the list of unsupported chains - count(case ss.synced when true then 1 else 1 end) as total, - count(case ss.synced when false then 1 else null end) as to_sync, - count(case ss.synced when true then 1 else null end) as synced - from sourcify_sync ss - where 1=1 - group by ss.chain_id - order by chain_id asc; - ``` - - The script will output a verification per second insight. The verification speed is a good compromise between the amount of concurrent requests sent for each chain and the amount of requests the rpc can handle. If you see that you are receiving a lot of rpc errors decrease the number of concurrent verifications, if the verifications per second are too low, increase the number of concurrent verifications. You can set the number of concurrent verifications using the `--limit` parameter. - - When the script was successful, the `sourcify-server-migration` instance can be stopped. - -### **4. Start using the new deprecated repositoryV1** - -- While the migration script was running on `sourcify-server-migration`, new matches coming from `sourcify-server` got saved in Database, RepositoryV2 and _current_repositoryV1_ . - - - There are no issues for the Database and RepositoryV2 because `sourcify-server-migration` and `sourcify-server` are writing in the same database/fs - - There is a complication regarding RepositoryV1. `sourcify-server` is storing new matches into the _current_repositoryV1_, while `sourcify-server-migration` in the _new_repositoryV1_. - - A procedure is needed to transfer all the contracts in _current_repositoryV1_ that were saved after RepositoryV2 and Database were enabled on `sourcify-server`. - - - `sourcify-server` stored the matches in `sourcify_sync` with the status set to `perfect` instead of `perfect_match` and `partial` instead of `partial_match`. We can use this information to list all the contracts that need to be transferred from _current_repositoryV1_ to _new_repositoryV1._ - - Extract all the contracts described in the previous point from the database in a `rows.txt` file. For each row output the path in the repository. You can use the following SQL query: - ```sql - SELECT - '/contracts/' || CASE WHEN match_type = 'perfect' THEN 'full_match' - WHEN match_type = 'partial' THEN 'partial_match' - ELSE 'partial_match' - END || '/' || chain_id || '/' || convert_from(address, 'UTF8') as path - FROM sourcify_sync - WHERE match_type = 'perfect' OR match_type = 'partial' - order by id desc; - ``` - - The `rows.txt` file will look like this: - ``` - /contracts/partial_match/11155111/0x6C5f9712ee69E76908D3E3Ef62Ce33c04d748f61 - /contracts/full_match/11155111/0x1fDe294eA2C5A81bC24fa117994436A0B30A8F12 - /contracts/partial_match/8453/0x85b1180AfC0c45fb074Baee447B8F4aBb5BF84A9 - /contracts/full_match/11155111/0x4bf0CE51d78041103eCAA6281503546D00721A2e - ... - ``` - - Use the following script to copy all the contracts from _current_repositoryV1_ to _new_repositoryV1_: - - ```bash - #!/bin/bash - - # Path to the file containing the lines - FILE_PATH="./rows.txt" - - # Count the total number of lines in the file - total_lines=$(wc -l < "$FILE_PATH") - - echo "Total files to process: $total_lines" - - # Variable to keep track of processed lines - processed_lines=0 - - # Read each line from the file - while IFS= read -r line - do - # Update processed lines count - ((processed_lines++)) - - # Construct source and destination paths - src="/home/app/repository_current$line" - dest="/home/app/repository$line" - - # Create the destination directory structure - mkdir -p "$(dirname "$dest")" - - # Copy the files from source to destination without overwriting existing files - cp -rn "$src" "$dest" - - # Calculate and display the percentage of completion - percent=$(($processed_lines * 100 / $total_lines)) - echo "Processed $processed_lines / $total_lines files ($percent%)" - done < "$FILE_PATH" - ``` - - In the script, modify `src` to match your _current_repositoryV1_ path, and `dest` to match your _new_repositoryV1_ path. - -- In your configuration file inside `services/server/src/config`, switch the RepositoryV1 config to the _new_repositoryV1_’s path. - - ```json - repositoryV1: { - path: "/home/app/repositoryV1", - ... - } - ``` - -- Restart `sourcify-server` - -## What’s next - -We are still working on finding solutions for the following points: - -- **Find a solution for unsupported chains contracts:** we migrated all the contracts that could be re-verified. Unfortunately, contracts from non-supported chains cannot be verified because the on-chain’s bytecode is no longer served by RPCs. This is currently work-in-progress and will come soon. -- **Using the DB as the source of truth**: after the migration the database will be used only as a storing service, the repository APIs are still using RepositoryV1. This is currently work-in-progress and will come soon. diff --git a/docs/3. Explore the Verified contracts data/1. Repo UI.mdx b/docs/3. Explore the Verified contracts data/1. Repo UI.mdx new file mode 100644 index 0000000..786b03f --- /dev/null +++ b/docs/3. Explore the Verified contracts data/1. Repo UI.mdx @@ -0,0 +1,90 @@ +--- +id: repo-ui +slug: /repo-ui +title: Repo UI +--- + +import annotatedBytecode from "./annotated-bytecode.png"; +import sources from "./sources.png"; +import storageLayout from "./storage-layout.png"; + +The Repository UI is Sourcify's web interface for browsing verified contracts. Use it to inspect contract details and artifacts across supported chains. + +What you can do: + +- Search by contract address and network. +- Inspect compilation settings, metadata, and match status. +- Browse/download source files and ABI directly in the browser. + +## Features + +Every contract's repository page shows compilation settings, ABI, metadata, and onchain information, plus the advanced views below. + +
+ Annotated bytecode +
Annotated bytecode
+
+ Annotated bytecode views (creation and runtime) highlight transformations + and signatures. +
+
+ +
+ Sources +
Sources
+
+ The Sources view provides a source explorer to navigate verified files + directly in the browser. +
+
+ +
+ Sources +
Storage Layout
+
+ The Storage Layout view displays variable slots. +
+
diff --git a/docs/4. repository/4. signature-database.mdx b/docs/3. Explore the Verified contracts data/2. Signatures database.md similarity index 88% rename from docs/4. repository/4. signature-database.mdx rename to docs/3. Explore the Verified contracts data/2. Signatures database.md index 3c456ea..651ba6a 100644 --- a/docs/4. repository/4. signature-database.mdx +++ b/docs/3. Explore the Verified contracts data/2. Signatures database.md @@ -1,4 +1,8 @@ -# Signature Database +--- +id: signature-database +slug: /signature-database +title: Signatures database +--- We also build an Ethereum signature database from our verified contracts. When you verify a contract, the function, event, and error signatures are extracted from its ABI and stored in the database. @@ -7,4 +11,4 @@ The data is stored in the same database as the verified contracts. The `signatur - **Schema**: Check [docs/repository/sourcify-database/#schema](/docs/repository/sourcify-database/#schema) for the schema. - **API**: Check [docs/api/](/docs/api/) for the API. - **Download**: You can download the related tables in Parquet format from [export.sourcify.dev](https://export.sourcify.dev). See [Download the Dataset](/docs/repository/download-dataset/) for more details. -- **Playground**: Visit [4byte.sourcify.dev](https://4byte.sourcify.dev) to search for signatures. \ No newline at end of file +- **Playground**: Visit [4byte.sourcify.dev](https://4byte.sourcify.dev) to search for signatures. diff --git a/docs/3. Explore the Verified contracts data/3. Dataset Playground.mdx b/docs/3. Explore the Verified contracts data/3. Dataset Playground.mdx new file mode 100644 index 0000000..6548e31 --- /dev/null +++ b/docs/3. Explore the Verified contracts data/3. Dataset Playground.mdx @@ -0,0 +1,36 @@ +--- +id: dataset-playground +slug: /dataset-playground +title: AI Dataset Playground +--- + +import datasetPlayground from "./dataset-playground.png"; + +The [AI Dataset Playground](https://sourcify.dev/#dataset-playground) allows users to explore the [Sourcify BigQuery dataset](http://localhost:3000/docs/bigquery/) using natural language. + +
+ Dataset playground +
+ The dataset playground at + [sourcify.dev](https://sourcify.dev/#dataset-playground) +
+
diff --git a/docs/4. repository/2. download-dataset.mdx b/docs/3. Explore the Verified contracts data/4. Download the dataset.md similarity index 98% rename from docs/4. repository/2. download-dataset.mdx rename to docs/3. Explore the Verified contracts data/4. Download the dataset.md index 85781fb..b205264 100644 --- a/docs/4. repository/2. download-dataset.mdx +++ b/docs/3. Explore the Verified contracts data/4. Download the dataset.md @@ -1,4 +1,8 @@ -# Download the Dataset +--- +id: dataset-download +slug: /repository/download-dataset/ +title: Download the dataset +--- :::warning @@ -43,11 +47,10 @@ aws s3 sync s3://sourcify-production-parquet-export/v2/ ./sourcify-dataset --end The `sourcify_matches` table is the only table that is not append-only and can be updated in the underlying Sourcify Database. Therefore, its rows are ordered by `updated_at` timestamps when exported. -This means that rows with the same `id` may appear multiple times in the export files. +This means that rows with the same `id` may appear multiple times in the export files. When working with the `sourcify_matches` table from the export, please only consider the row with the most recent `updated_at` for each `id` as the current state. For importing the `sourcify_matches` parquet files into a database, an **upsert** operation should be used. - ### Working with Parquet Files Once downloaded, you can query and analyze Parquet files using various tools and libraries. Here are some popular options to give you a head start: diff --git a/docs/3. Explore the Verified contracts data/6. BigQuery.md b/docs/3. Explore the Verified contracts data/6. BigQuery.md new file mode 100644 index 0000000..f5a55cc --- /dev/null +++ b/docs/3. Explore the Verified contracts data/6. BigQuery.md @@ -0,0 +1,11 @@ +--- +id: bigquery +slug: /bigquery +title: BigQuery +--- + +We provide a public BigQuery dataset for convenient querying and exploration: + +[Sourcify BigQuery dataset](https://console.cloud.google.com/bigquery/analytics-hub/exchanges/projects/1019539084286/locations/europe-west1/dataExchanges/sourcify_19a0c79ef3a/listings/sourcify_19a0c7d0be2?project=tranquil-petal-125711) + +The dataset is updated continuously as new contracts are verified. You need a Google account to access it. diff --git a/docs/3. Explore the Verified contracts data/_category_.json b/docs/3. Explore the Verified contracts data/_category_.json new file mode 100644 index 0000000..0d44758 --- /dev/null +++ b/docs/3. Explore the Verified contracts data/_category_.json @@ -0,0 +1,3 @@ +{ + "collapsed": false +} diff --git a/docs/3. Explore the Verified contracts data/annotated-bytecode.png b/docs/3. Explore the Verified contracts data/annotated-bytecode.png new file mode 100644 index 0000000..33e9a49 Binary files /dev/null and b/docs/3. Explore the Verified contracts data/annotated-bytecode.png differ diff --git a/docs/3. Explore the Verified contracts data/dataset-playground.png b/docs/3. Explore the Verified contracts data/dataset-playground.png new file mode 100644 index 0000000..9a50c56 Binary files /dev/null and b/docs/3. Explore the Verified contracts data/dataset-playground.png differ diff --git a/docs/3. Explore the Verified contracts data/sources.png b/docs/3. Explore the Verified contracts data/sources.png new file mode 100644 index 0000000..addb8e4 Binary files /dev/null and b/docs/3. Explore the Verified contracts data/sources.png differ diff --git a/docs/3. Explore the Verified contracts data/storage-layout.png b/docs/3. Explore the Verified contracts data/storage-layout.png new file mode 100644 index 0000000..2450265 Binary files /dev/null and b/docs/3. Explore the Verified contracts data/storage-layout.png differ diff --git a/docs/3. Verification/1.howToVerify.mdx b/docs/3. Verification/1.howToVerify.mdx deleted file mode 100644 index 1535d17..0000000 --- a/docs/3. Verification/1.howToVerify.mdx +++ /dev/null @@ -1,197 +0,0 @@ ---- -id: how-to-verify -title: How to Verify Contracts -slug: /how-to-verify ---- - -import chainAddress from "./chain-address.png" -import verificationMethods from "./methods.png" -import importSources from "./import-sources.png" -import jobError from "./job-error.png" -import bytecodeDiff from "./bytecode-diff.png" -import buildInfo from "./build-info.png" -import TOCInline from '@theme/TOCInline'; - - -# How to Verify Contracts - -There are multiple ways to verify a contract on Sourcify. Depending on your use case, you can choose the best way. - -Some of these methods are using the legacy Sourcify API. This will be deprecated in the near future. Use Sourcify APIv2 instead (see [APIv2 docs](/docs/api)). - -With APIv2 when you submit a verification, this will create a verification job. You can then check the status of the job and get the verification result. See one example in the UI [here](https://verify.sourcify.dev/jobs/f4ed8a9b-c852-4ac6-ac99-bd71ffb01305). - -**Methods:** - - - -## verify.sourcify.dev UI - -You can verify using our new verification UI at https://verify.sourcify.dev. See the Github repo [here](https://github.com/sourcifyeth/verify.sourcify.dev). - -First choose the chain you want to verify the contract on and provide the contract address. -
- Chain and address -
- -This will automatically check if the contract is verified on Sourcify on the same chain and all other supported chains. - -### Importing a contract - -You can import a verified contract from different sources: - -- **Etherscan**: Enter the chain and contract address to import the contract. You need an Etherscan API key to use this feature. Open the "Settings" on top right to add your API key. API keys are used locally in the browser and are not sent to Sourcify servers. -
- Import sources -
- -### Verification Methods - -For both Solidity and Vyper contracts you can choose between the following verification methods: - -- **Single file**: Provide single .sol or .vy file, alongside compiler settings. -- **Multiple files**: Provide multiple .sol or .vy files, alongside compiler settings. -- (preferred) **Standard JSON**: Provide a [standard JSON input file](https://docs.soliditylang.org/en/v0.8.30/using-the-compiler.html#input-description). - -Prefer using the standard JSON or framework commands directly (see below) to maintain the full compilation and avoid any changes. Different compiler settings can break the verification. Flattening the sources also breaks the metadata hashes and you won't be able to get an ["Exact Match"](/docs/exact-match-vs-match). - -Additionally the following option is available for Solidity contracts: -- **Metadata.json**: Provide a [metadata.json file](/docs/metadata) to verify the contract in the first file input field. Provide all the source files in the second file input field. This option checks the source hashes in the metadata.json provided and compares them with the hashes of the source files provided before submitting the verification. - -
-Verification methods -
Available verification methods
-
- -### Contract Identifier - -From the files and contracts provided, we will ask you to select the contract you want to verify. Ie. if you've written a contract called `MyContract.sol` using other contracts `Ownable.sol` and `ERC20.sol`, you will be asked to select `MyContract.sol` as the contract to verify. - -The format of the contract identifier is path:contractName e.g. -``` -path/to/contractName.sol:contractName -``` - -The verification UI will parse the files and give you a list of contracts to choose from. You can always input the contract identifier manually. - -### (Optional) Creation Transaction Hash - -Sourcify needs to find the transaction the contract was deployed in to get the creation bytecode. We get this information from different sources like block explorers (see [here](https://github.com/argotorg/sourcify/blob/staging/services/server/src/sourcify-chains-default.json)). You can still provide a creation transaction hash manually if you want to, or if Sourcify fails to verify via creation bytecode. - -### Hardhat/Foundry build-info files - -If you are using Hardhat or Foundry, the best way to verify is to directly use the CLI of the frameworks (see the commands below). However, it is also possible to use the Sourcify build-info files to verify. - -The build-info files are generated by the frameworks and contain the compilation information. This can be parsed and used by the verify.sourcify.dev UI to verify the contract. - -You can find the build-info files in: -- Hardhat: `artifacts/build-info/.json` -- Foundry: `out/build-info/.json` - -
- Build info -
Build info files toggle
-
- -Click on the toggle to "Upload build-info file" and select the build-info file. - -## Verification Job - -Once you've provided all the information, you can submit the verification. This will create a verification job. You can see the job in the UI and get the verification result. - -If the verification fails you can see the error message and why it failed. - -
- Job error -
Example of a verification job that failed
-
- -You can see the diffs of the bytecodes if the compilation succeeds but the bytecodes did not match. - -
- Bytecode diff -
Bytecode diffs
-
- -## Foundry (legacy API) - -Foundry natively supports Sourcify verification. You can verify contracts with the additional verify flags in Forge. - -(This is still using the legacy Sourcify API) - -```bash title="Deploy and verify a contract" -forge create --rpc-url \ - --constructor-args "ForgeUSD" "FUSD" 18 1000000000000000000000 \ - --private-key \ - src/MyToken.sol:MyToken \ - --verify \ - --verification-provider sourcify - --verifier-url https://localhost:5555 # optional, defaults to https://sourcify.dev/server/ -``` - -```bash title="Verify an already deployed (and compiled) contract" -forge verify-contract 0x55f7d4279CE387067f12561e7E0c194f5186cFba \ - src/MyToken.sol:MyToken \ - --chain-id 11155111 \ - --verifier sourcify - --verifier-url https://localhost:5555 # optional, defaults to https://sourcify.dev/server/ -``` - -```bash title="Check if a contract is verified" -forge verify-check 0x55f7d4279CE387067f12561e7E0c194f5186cFba \ - --chain-id 11155111 \ - --verifier sourcify -``` - -## Hardhat (legacy API) - -### hardhat-verify plugin - -The Hardhat plugin [@nomicfoundation/hardhat-verify](https://hardhat.org/hardhat-runner/plugins/nomicfoundation-hardhat-verify#verifying-on-sourcify) has official Sourcify support. - -(This is still using the legacy Sourcify API) - -In order to enable Sourcify support, add this to `hardhat.config.js`: - -```js title="hardhat.config.js" -sourcify: { - enabled: true, - // Optional: specify a different Sourcify server - apiUrl: "https://sourcify.dev/server", - // Optional: specify a different Sourcify repository - browserUrl: "https://repo.sourcify.dev", -} -``` - -After deploying your contracts you can verify on Sourcify with: - -```bash title="Verify a contract" -npx hardhat verify --network mainnet DEPLOYED_CONTRACT_ADDRESS -``` - -## Remix - -You can use the "unified verification plugin" written by the Sourcify team to verify on multiple verifiers on Remix. - -1. Click on the "Plugins" tab in bottom of the sidebar and search for "Contract Verification" -2. Enable the plugin "Contract Verification" -3. Click on the "Contract Verification" tab in the sidebar -5. Enter the contract address and select a chain -6. Select the contract to verify (you need to have compiled a contract on Remix first) -7. By default all verifiers are enabled. For Etherscan you need to add your API key in the "Settings" tab -8. You can see the verification result in the "Receipts" tab - -
- -
-## API - -See the [API docs](/docs/api) for more information. - -## Other Sourcify wrappers and tools - -- [sourcify-go](https://github.com/unpackdev/sourcify-go) - -Do you have a tool to be listed? Let us know! diff --git a/docs/3. Verification/6. localNetwork.md b/docs/3. Verification/6. localNetwork.md deleted file mode 100644 index c8457f4..0000000 --- a/docs/3. Verification/6. localNetwork.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -id: local-test-chain -title: Verification on Local Test Chain -slug: /local-test-chain ---- - -# Verification on Local Test Chain - -When running locally i.e. `NODE_ENV !== production`, Sourcify adds the chainIds for the Ganache (1337) and Hardhat Network (31337) by default. - -If you are running a different local chain, you can add it to Sourcify manually. - -First, find the `chainId` of your local chain. For [Hardhat network](https://hardhat.org/hardhat-network/reference/#chainid) default is `31337`. For [Ganache](https://github.com/trufflesuite/ganache) it is `1337`. If you don't know the chain id of your network, you can send the [`net_version` RPC call](https://eth.wiki/json-rpc/API#net_version) and look at the `result` field: - -``` -$ curl localhost:8545 -X POST --data '{"jsonrpc":"2.0","method":"net_version","params":[],"id":67}' - -{"jsonrpc":"2.0","id":67,"result":"31337"} -``` - -Similar to [adding EVM chains](/docs/chain-support) add the chain to [`sourcify-chains-default.json`](https://github.com/argotorg/sourcify/blob/staging/services/server/sourcify-chains-default.json), or to override all other chains you can simply create a new file `sourcify-chains.json` in the same directory. For example, to add the Hardhat Network: - -```js -... - ... - "31337": { - "sourcifyName": "Hardhat Network", - "supported": true, - "rpc": ["http://localhost:8545"], - }, - ... -``` - -Now you can deploy and verify contracts on the testnet. Don't forget to rebuild the repo for changes to take effect: - -``` -npx lerna run build -``` - -Start the server - -``` -npm run server:start -``` - -Start the UI - -``` -npm run ui:start -``` diff --git a/docs/3. Verification/9. vyper.md b/docs/3. Verification/9. vyper.md deleted file mode 100644 index 5bfc94e..0000000 --- a/docs/3. Verification/9. vyper.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -id: vyper -title: Vyper verification -slug: /vyper ---- - -# Vyper verification - -Sourcify supports Vyper verification via the `/v2/verify` API endpoint, you can find more information in the [APIs documentation](/docs/api/). - -## Vyper Feature Support - -- 🟢 **Match**: Sourcify uses the metadata hash as a compilation fingerprint (you can read more about it [here](/docs/exact-match-vs-match)), since Vyper doesn't support metadata, every Vyper match will result in a simple Match (no Exact Match). Starting from version 0.4.1 Vyper implements a similar concept called [integrity](https://docs.vyperlang.org/en/stable/compiling-a-contract.html#integrity-hash), but Sourcify doesn't support it yet. -- 🟢 **Database transformations**: For more details, see the [Sourcify Database section](/docs/repository/sourcify-database/). Currently, Vyper verification supports `cborAuxdata`, `immutable` and `constructorArguments` transformations. -- 🟢 **Generated metadata**: Sourcify generates a metadata file for Vyper contracts, but it was used only for compatibility with the legacy API, since everything was based on the Solidity metadata.json. -- 🟡 **IPFS support**: Sourcify automatically uploads all verified contract's sources to IPFS. Even if the files are on IPFS, there is no link to them in the Vyper contract's bytecode. This is because Vyper doesn't include the metadata IPFS CID in the bytecode's auxdata. -- 🔴 **Exact Match**: Not possible due to lack of metadata support. -- 🔴 **Monitoring**: Monitor listens for `create` operations on many chains. It attempts to fetch the sources of deployed contracts from IPFS using the metadata embedded in the bytecodes. Since Vyper doesn't support metadata, Sourcify can't fetch the sources from IPFS. diff --git a/docs/3. Verification/9.1. yul.md b/docs/3. Verification/9.1. yul.md deleted file mode 100644 index 510ec2c..0000000 --- a/docs/3. Verification/9.1. yul.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -id: yul -title: Yul verification -slug: /yul -sidebar_position: 9 ---- - -# Yul verification - -Sourcify supports Yul verification via the `/v2/verify` API endpoint, you can find more information in the [APIs documentation](/docs/api/). - -## Yul Feature Support - -- 🟢 **Match**: Sourcify uses the metadata hash as a compilation fingerprint (you can read more about it [here](/docs/exact-match-vs-match)), since Yul doesn't support metadata, every Yul match will result in a simple Match (no Exact Match). -- 🟢 **Generated metadata**: Sourcify generates a metadata file for Yul contracts, but it was used only for compatibility with the legacy API, since everything was based on the Solidity metadata.json. -- 🔴 **ABI**: Yul compilation doesn't generate ABI. -- 🔴 **Exact Match**: Not possible due to lack of metadata support. diff --git a/docs/3. Verification/etherscanInstancesTable.tsx b/docs/3. Verification/etherscanInstancesTable.tsx deleted file mode 100644 index 2596b19..0000000 --- a/docs/3. Verification/etherscanInstancesTable.tsx +++ /dev/null @@ -1,74 +0,0 @@ -import React, { useEffect, useState } from "react"; -import LoadingOverlay from "../../src/components/LoadingOverlay"; - -export default function EtherscanInstancesTable() { - const [etherscanInstances, setEtherscanInstances] = useState(); - const [error, setError] = useState(""); - - useEffect(() => { - fetch("https://sourcify.dev/server/chains") - .then((res) => res.json()) - .then((sourcifyChains) => { - const etherscanChains = sourcifyChains.filter( - (chain) => !!chain.etherscanAPI - ); - setEtherscanInstances(etherscanChains); - }) - .catch((err) => - setError( - "Error fetching chains from the Sourcify server\n\n" + err.message - ) - ); - }, []); - - if (error) { - return ( -
-

{error}

-
- ); - } - if (!etherscanInstances) { - return ( -
- -
- ); - } - - const rows = etherscanInstances.map((chain, i) => { - const etherscanLink = chain.etherscanAPI - .replace("api.", "") - .replace("api-", ""); - return ( - - {chain.title || chain.name} - -
- {chain.chainId} -
- - - - {etherscanLink} - - - - ); - }); - - return ( -
- - - - - - - - - {rows} -
Chain NameChain IDEtherscan Instance Link
-
- ); -} diff --git a/docs/3. Verification/etherscanScreenshot.png b/docs/3. Verification/etherscanScreenshot.png deleted file mode 100644 index 29288b6..0000000 Binary files a/docs/3. Verification/etherscanScreenshot.png and /dev/null differ diff --git a/docs/5. API.mdx b/docs/4. Integrate Sourcify/1. API.md similarity index 83% rename from docs/5. API.mdx rename to docs/4. Integrate Sourcify/1. API.md index 02bc34c..fefea29 100644 --- a/docs/5. API.mdx +++ b/docs/4. Integrate Sourcify/1. API.md @@ -4,7 +4,7 @@ title: API slug: /api --- -import SwaggerContainer from "../src/components/SwaggerContainer"; +import SwaggerContainer from "../../src/components/SwaggerContainer"; # API Documentation @@ -12,7 +12,6 @@ import SwaggerContainer from "../src/components/SwaggerContainer"; - [Server API Documentation](#server-api-documentation) - [4byte (Signature) Service API Documentation](#4byte-signature-service-api-documentation) -- [Other Endpoints](#other) ## Server API Documentation diff --git a/docs/4. Integrate Sourcify/2. Typescript libraries/1. lib-sourcify.mdx b/docs/4. Integrate Sourcify/2. Typescript libraries/1. lib-sourcify.mdx new file mode 100644 index 0000000..7fee302 --- /dev/null +++ b/docs/4. Integrate Sourcify/2. Typescript libraries/1. lib-sourcify.mdx @@ -0,0 +1,14 @@ +--- +id: lib-sourcify +slug: /lib-sourcify +title: lib-sourcify +hide_title: true +--- + +:::info +lib-sourcify is used to directly verify contracts, it's NOT a wrapper around the Sourcify API. lib-sourcify works also in the browser, allowing source code verification to be executed in the user environment. +::: + +import EmbedReadmeLibSourcifyPackage from "./EmbedReadmeLibSourcifyPackage"; + + diff --git a/docs/4. Integrate Sourcify/2. Typescript libraries/2. bytecode-utils.md b/docs/4. Integrate Sourcify/2. Typescript libraries/2. bytecode-utils.md new file mode 100644 index 0000000..8e19f88 --- /dev/null +++ b/docs/4. Integrate Sourcify/2. Typescript libraries/2. bytecode-utils.md @@ -0,0 +1,10 @@ +--- +id: bytecode-utils +slug: /bytecode-utils +title: bytecode-utils +hide_title: true +--- + +import EmbedReadmeBytecodeUtilsPackage from "./EmbedReadmeBytecodeUtilsPackage" + + diff --git a/docs/4. Integrate Sourcify/2. Typescript libraries/3. compilers.md b/docs/4. Integrate Sourcify/2. Typescript libraries/3. compilers.md new file mode 100644 index 0000000..95c5b70 --- /dev/null +++ b/docs/4. Integrate Sourcify/2. Typescript libraries/3. compilers.md @@ -0,0 +1,10 @@ +--- +id: compilers +slug: /compilers +title: compilers +hide_title: true +--- + +import EmbedReadmeCompilersPackage from "./EmbedReadmeCompilersPackage" + + diff --git a/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeBytecodeUtilsPackage.jsx b/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeBytecodeUtilsPackage.jsx new file mode 100644 index 0000000..c06203d --- /dev/null +++ b/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeBytecodeUtilsPackage.jsx @@ -0,0 +1,19 @@ +import EmbedReadme from "../../EmbedReadme"; +import React from "react"; +const repo = "argotorg/sourcify"; +const branch = "master"; +const readmePath = "/packages/bytecode-utils/README.md"; +const rawBaseUrl = "https://raw.githubusercontent.com"; +const pageBaseUrl = "https://github.com"; + +export default function EmbedReadmeRepoSourcifyDev() { + return ( + + ); +} diff --git a/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeCompilersPackage.jsx b/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeCompilersPackage.jsx new file mode 100644 index 0000000..187d7dd --- /dev/null +++ b/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeCompilersPackage.jsx @@ -0,0 +1,19 @@ +import EmbedReadme from "../../EmbedReadme"; +import React from "react"; +const repo = "argotorg/sourcify"; +const branch = "master"; +const readmePath = "/packages/compilers/README.md"; +const rawBaseUrl = "https://raw.githubusercontent.com"; +const pageBaseUrl = "https://github.com"; + +export default function EmbedReadmeRepoSourcifyDev() { + return ( + + ); +} diff --git a/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeLibSourcifyPackage.jsx b/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeLibSourcifyPackage.jsx new file mode 100644 index 0000000..aedcda6 --- /dev/null +++ b/docs/4. Integrate Sourcify/2. Typescript libraries/EmbedReadmeLibSourcifyPackage.jsx @@ -0,0 +1,19 @@ +import EmbedReadme from "../../EmbedReadme"; +import React from "react"; +const repo = "argotorg/sourcify"; +const branch = "master"; +const readmePath = "/packages/lib-sourcify/README.md"; +const rawBaseUrl = "https://raw.githubusercontent.com"; +const pageBaseUrl = "https://github.com"; + +export default function EmbedReadmeRepoSourcifyDev() { + return ( + + ); +} diff --git a/docs/4. Integrate Sourcify/_category_.json b/docs/4. Integrate Sourcify/_category_.json new file mode 100644 index 0000000..0d44758 --- /dev/null +++ b/docs/4. Integrate Sourcify/_category_.json @@ -0,0 +1,3 @@ +{ + "collapsed": false +} diff --git a/docs/4. repository/3. file-repositories.mdx b/docs/4. repository/3. file-repositories.mdx deleted file mode 100644 index 1d2c9a4..0000000 --- a/docs/4. repository/3. file-repositories.mdx +++ /dev/null @@ -1,143 +0,0 @@ -import TotalRepoSize from "./TotalRepoSize" - -# File Repositories - -:::danger Deprecation Notice -The file repositories are **deprecated** and no longer supported. The [Sourcify Database](/docs/repository/sourcify-database/) serves as the source of truth now. - -Only the IPFS pinning service still uses the logical structure of RepositoryV2 for uploading files to IPFS. - -For custom Sourcify instances, we recommend migrating to APIv2 and the [Database](/docs/repository/sourcify-database/) as the main storage backend. See the [migration guide](/docs/database-migration). -::: - -This page describes the `RepositoryV1` and `RepositoryV2`, which are file systems (deprecated). - -## Table of Contents - -- [RepositoryV1 vs RepositoryV2](#repositoryv1-vs-repositoryv2) - - [RepositoryV1](#repositoryv1) - - [RepositoryV2](#repositoryv2) -- [Download](#download) - - -## RepositoryV1 vs RepositoryV2 - -### RepositoryV1 -RepositoryV1 is the legacy storage backend for files. It is simply a file system based on how file paths are given in the [Solidity metadata](/docs/metadata). file. - -An [example metadata](https://repo.sourcify.dev/contracts/full_match/1/0x801f3983c7baBF5E6ae192c84E1257844aDb4b4D/metadata.json) source file paths look like this for the "full_match" contract `0x801f3983c7baBF5E6ae192c84E1257844aDb4b4D` on Ethereum Mainnet (1): -```json -{ - "sources": { - "erc20/IERC20.sol": { - "keccak256": "0xa38ec4e151e4d397d05bdfb94e6e4eb91e57a9fca3bc1c655289a4adf31a58fa", - "license": "MIT", - "urls": [ - "bzz-raw://312e850e36efbf0f2450896c213b23dc0a28150e051bcbf933a8b9211627c44b", - "dweb:/ipfs/QmWsyisPjDwTJrTMhsGZa4JHiCS63mWfsyVQKbaijWGdmK" - ] - }, - "erc20/airdrop.sol": { - "keccak256": "0xea27a3e2c4179a064caf9fe9a198addd526fd1d1ea467ea474a0c069e6eac957", - "urls": [ - "bzz-raw://6a86bc69b99876768bdbddba504410cf60b33681e1203a36d98840bf2ab8a42b", - "dweb:/ipfs/QmRZSqNfAPduoPoUJ6BM4NpBTbTKBqg5Mz5YBNpaUz4TfQ" - ] - } - }, -} -``` - -These files will be like below ([see in repo](https://repo.sourcify.dev/contracts/full_match/1/0x801f3983c7baBF5E6ae192c84E1257844aDb4b4D/)): -``` -contracts/full_match/1/0x801f3983c7baBF5E6ae192c84E1257844aDb4b4D/ -├── metadata.json -└── sources/ - └── erc20/ - ├── IERC20.sol - └── airdrop.sol -``` -The problem with this is the part `"erc20/airdrop.sol"` is not necessarily a valid file path but a ["source unit name"](https://docs.soliditylang.org/en/v0.8.27/path-resolution.html#virtual-filesystem:~:text=assigned%20a%20unique-,source%20unit%20name,-which%20is%20an) in Solidity, i.e. arbitrary strings. This may cause issues on file systems as well as when pinning to IPFS. - -### RepositoryV2 - -RepositoryV2 is the format where we normalize the file names with their keccak256 hashes (source files must have a `keccak256` field in the metadata). So the example above would look like this: - -``` -contracts/full_match/1/0x801f3983c7baBF5E6ae192c84E1257844aDb4b4D/ -├── metadata.json -└── sources/ - └── 0xa38ec4e151e4d397d05bdfb94e6e4eb91e57a9fca3bc1c655289a4adf31a58fa - └── 0xea27a3e2c4179a064caf9fe9a198addd526fd1d1ea467ea474a0c069e6eac957 -``` - -The files are exactly the same so their IPFS hashes will not change, and you can look up the metadata file to find the original path-alike source unit names. - -## IPFS - -The sources of all verified contracts are pinned on IPFS. The logical structure of RepositoryV2 serves as the basis for uploading to IPFS. Files can be accessed via their individual CIDs (e.g. [`QmVij3h9z536ZG5cRpUmTfdoN9KR1Xp4ix2P7to9dPHgE5`](https://ipfs.io/ipfs/QmVij3h9z536ZG5cRpUmTfdoN9KR1Xp4ix2P7to9dPHgE5)). - -Unfortunately, publishing under IPNS is temporarily disabled. This is because of the difficulty of managing the whole filesystem over IPFS (with MFS etc.) and updating the IPNS regularly. - - -## Download - -:::danger No New Exports -Following deprecating the filesystem based repositories, **we no longer publish new exports**. We recommend resorting to the [Parquet exports](/docs/repository/download-dataset/) instead. - -You can still download the existing export for a while. Double check the date of the export in the manifest file. If you need these exports please reach out to us. -::: - -We compress the **RepositoryV2** weekly and publish on Cloudflare R2 under https://repo-backup.sourcify.dev ( `.dev` redirects to `.app` domain, which also belongs to Sourcify). - - - -[repo-backup.sourcify.dev](https://repo-backup.sourcify.dev) will redirect to a `manifest.json` file: - -
-manifest.json - -```json -{ - "description": "Manifest file for when the Sourcify file repository was uploaded", - "timestamp": 1726030203254, - "dateStr": "2024-09-11T04:50:03.254904Z", - "files": [ - { - "path": "/sourcify-repository-2025-01-06T03-00-24/full_match.1.00.tar.gz" - "sizeInBytes": 2097152000 - }, - { - "path": "/sourcify-repository-2025-01-06T03-00-24/full_match.1.01.tar.gz" - "sizeInBytes": 800472503 - }, - ... - ] -} -``` -
- -The files are split by match type, chain and the first byte of the contract address. -E.g. `/full_match.1.2F.tar.gz` will contain all the full_match contracts for chain 1 starting with `0x2F`. - -You can download all files in the `files` array and unzip them: - -1. Download the manifest file (`-L` to follow redirects): - ```bash - curl -L -O https://repo-backup.sourcify.dev/manifest.json - ``` - -2. Extract file paths and download each file: - ```bash - jq -r '.files[].path' manifest.json | xargs -I {} curl -L -O https://repo-backup.sourcify.dev/{} - ``` - -3. Concatenate the downloaded parts: - ```bash - cat sourcify-repository-*.part.gz.* > sourcify-repository.gz - ``` - -4. Unzip the concatenated file: - ```bash - tar -xvzf sourcify-repository.gz - ``` diff --git a/docs/4. repository/TotalRepoSize.tsx b/docs/4. repository/TotalRepoSize.tsx deleted file mode 100644 index 3350f4a..0000000 --- a/docs/4. repository/TotalRepoSize.tsx +++ /dev/null @@ -1,52 +0,0 @@ -import React, { useState, useEffect } from "react"; -import LoadingOverlay from "../../src/components/LoadingOverlay"; - -const RepositoryStats = () => { - const [isLoading, setIsLoading] = useState(true); - const [totalSize, setTotalSize] = useState(0); - const [timestamp, setTimestamp] = useState(""); - - useEffect(() => { - const fetchStats = async () => { - const manifestUrl = "https://repo-backup.sourcify.app/manifest.json"; - - try { - const manifestResponse = await fetch(manifestUrl); - const manifestData = await manifestResponse.json(); - - const totalSizeBytes = manifestData.files.reduce((acc, file) => acc + file.sizeInBytes, 0); - const totalSizeGB = totalSizeBytes / (1024 * 1024 * 1024); // Convert to GB - - setTotalSize(totalSizeGB); - - const date = new Date(manifestData.timestamp); - const formattedDate = date - .toUTCString() - .replace(/^[A-Za-z]+, /, "") - .replace(/:\d{2} /, " "); - setTimestamp(formattedDate); - } catch (error) { - console.error("Error fetching manifest:", error); - } finally { - setIsLoading(false); - } - }; - - fetchStats(); - }, []); - - if (isLoading) { - return ; - } - - return ( -
-

- As of {timestamp} the compressed size of the repository files is:{" "} - {totalSize.toFixed(2)} GB -

-
- ); -}; - -export default RepositoryStats; diff --git a/docs/4. repository/index.mdx b/docs/4. repository/index.mdx deleted file mode 100644 index 8599d19..0000000 --- a/docs/4. repository/index.mdx +++ /dev/null @@ -1,13 +0,0 @@ -# Contract Dataset - -Sourcify stores all verified contracts in multiple storage backends and gives multiple options to access the dataset. In short, there are the following options: - -- **Sourcify Database**: Sourcify's source of truth, a [postgres database](/docs/repository/sourcify-database). Accessible via the [API](/docs/api/) and the [Repo UI](https://repo.sourcify.dev/). -- **Verifier Alliance Database**: Shared database with other verification services. See the [Verifier Alliance](https://verifieralliance.org/) website for more info. -- **BigQuery**: For convenience, the Sourcify dataset is uploaded to [BigQuery](/docs/repository/sourcify-database/#bigquery-datasets). -- **IPFS**: The sources of all verified contracts are pinned on [IPFS](/docs/repository/file-repositories/#ipfs). - - -## Download - -You can download the Sourcify database in Parquet format. Follow this guide: [Download the Dataset](/docs/repository/download-dataset/). diff --git a/docs/2. Running Sourcify/1-Server.mdx b/docs/6. Run your own Sourcify instance/1. Run server.md similarity index 53% rename from docs/2. Running Sourcify/1-Server.mdx rename to docs/6. Run your own Sourcify instance/1. Run server.md index 0353ca6..a3a4ecb 100644 --- a/docs/2. Running Sourcify/1-Server.mdx +++ b/docs/6. Run your own Sourcify instance/1. Run server.md @@ -1,8 +1,10 @@ --- -id: Server +id: run-server +title: Run server slug: /running-server +hide_title: true --- import EmbedReadmeServer from "./EmbedReadmeServer" - \ No newline at end of file + diff --git a/docs/2. Running Sourcify/5-Monitor.mdx b/docs/6. Run your own Sourcify instance/2. Run monitor.md similarity index 62% rename from docs/2. Running Sourcify/5-Monitor.mdx rename to docs/6. Run your own Sourcify instance/2. Run monitor.md index abd564f..227dc61 100644 --- a/docs/2. Running Sourcify/5-Monitor.mdx +++ b/docs/6. Run your own Sourcify instance/2. Run monitor.md @@ -1,8 +1,10 @@ --- id: Monitor slug: /running-monitor +title: Run monitor +hide_title: true --- import EmbedReadmeMonitor from "./EmbedReadmeMonitor" - \ No newline at end of file + diff --git a/docs/2. Running Sourcify/2-verify.sourcify.dev.mdx b/docs/6. Run your own Sourcify instance/3. Run verification UI.md similarity index 63% rename from docs/2. Running Sourcify/2-verify.sourcify.dev.mdx rename to docs/6. Run your own Sourcify instance/3. Run verification UI.md index 5ae6198..e12298a 100644 --- a/docs/2. Running Sourcify/2-verify.sourcify.dev.mdx +++ b/docs/6. Run your own Sourcify instance/3. Run verification UI.md @@ -1,8 +1,10 @@ --- +title: Run verification UI id: verify.sourcify.dev slug: /verify.sourcify.dev +hide_title: true --- import EmbedReadmeVerifySourcifyDev from "./EmbedReadmeVerifySourcifyDev" - \ No newline at end of file + diff --git a/docs/2. Running Sourcify/3-repo.sourcify.dev.mdx b/docs/6. Run your own Sourcify instance/4. Run repository UI.md similarity index 63% rename from docs/2. Running Sourcify/3-repo.sourcify.dev.mdx rename to docs/6. Run your own Sourcify instance/4. Run repository UI.md index 8cb073d..48bab5b 100644 --- a/docs/2. Running Sourcify/3-repo.sourcify.dev.mdx +++ b/docs/6. Run your own Sourcify instance/4. Run repository UI.md @@ -1,8 +1,10 @@ --- id: repo.sourcify.dev slug: /repo.sourcify.dev +title: Run repository UI +hide_title: true --- import EmbedReadmeRepoSourcifyDev from "./EmbedReadmeRepoSourcifyDev" - \ No newline at end of file + diff --git a/docs/2. Running Sourcify/6-Signature.mdx b/docs/6. Run your own Sourcify instance/5. Run 4bytes UI.md similarity index 63% rename from docs/2. Running Sourcify/6-Signature.mdx rename to docs/6. Run your own Sourcify instance/5. Run 4bytes UI.md index 8584eb0..6fd36a6 100644 --- a/docs/2. Running Sourcify/6-Signature.mdx +++ b/docs/6. Run your own Sourcify instance/5. Run 4bytes UI.md @@ -1,8 +1,10 @@ --- id: 4byte Service slug: /running-4byte +title: Run 4bytes UI +hide_title: true --- import EmbedReadme4byteService from "./EmbedReadme4byteService" - \ No newline at end of file + diff --git a/docs/2. Running Sourcify/7-Tests.md b/docs/6. Run your own Sourcify instance/6 Tests.md similarity index 100% rename from docs/2. Running Sourcify/7-Tests.md rename to docs/6. Run your own Sourcify instance/6 Tests.md diff --git a/docs/2. Running Sourcify/EmbedReadme4byteService.jsx b/docs/6. Run your own Sourcify instance/EmbedReadme4byteService.jsx similarity index 100% rename from docs/2. Running Sourcify/EmbedReadme4byteService.jsx rename to docs/6. Run your own Sourcify instance/EmbedReadme4byteService.jsx diff --git a/docs/2. Running Sourcify/EmbedReadmeMonitor.jsx b/docs/6. Run your own Sourcify instance/EmbedReadmeMonitor.jsx similarity index 100% rename from docs/2. Running Sourcify/EmbedReadmeMonitor.jsx rename to docs/6. Run your own Sourcify instance/EmbedReadmeMonitor.jsx diff --git a/docs/2. Running Sourcify/EmbedReadmeRepoSourcifyDev.jsx b/docs/6. Run your own Sourcify instance/EmbedReadmeRepoSourcifyDev.jsx similarity index 100% rename from docs/2. Running Sourcify/EmbedReadmeRepoSourcifyDev.jsx rename to docs/6. Run your own Sourcify instance/EmbedReadmeRepoSourcifyDev.jsx diff --git a/docs/2. Running Sourcify/EmbedReadmeServer.jsx b/docs/6. Run your own Sourcify instance/EmbedReadmeServer.jsx similarity index 100% rename from docs/2. Running Sourcify/EmbedReadmeServer.jsx rename to docs/6. Run your own Sourcify instance/EmbedReadmeServer.jsx diff --git a/docs/2. Running Sourcify/EmbedReadmeVerifySourcifyDev.jsx b/docs/6. Run your own Sourcify instance/EmbedReadmeVerifySourcifyDev.jsx similarity index 100% rename from docs/2. Running Sourcify/EmbedReadmeVerifySourcifyDev.jsx rename to docs/6. Run your own Sourcify instance/EmbedReadmeVerifySourcifyDev.jsx diff --git a/docs/97. Glossary.md b/docs/97. Glossary.md deleted file mode 100644 index 322ab99..0000000 --- a/docs/97. Glossary.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -id: glossary -title: Glossary -slug: /glossary ---- - -# Glossary - -Since Ethereum, EVM, and Solidity is full of not really well defined terms, we try to clear the ones we use here: - -### creation bytecode -The bytecode run by the EVM to deploy the smart contract. This will be wrap the *deployed bytecode* with prefixed instructions to deploy the contract, as well as suffixed constructor arguments in [ABI encoding](https://docs.soliditylang.org/en/v0.8.14/abi-spec.html#abi). - - -### deployed bytecode -The bytecode of a smart contract that will be deployed on-chain i.e. the value you'd get when calling [eth_getcode](https://www.quicknode.com/docs/ethereum/eth_getCode) on the contract. Note that with the contracts with [immutables](/docs/immutables), the deployed bytecode after recompilation will be different than the actually deployed contract. \ No newline at end of file