Modify doc scripts to copy agent-api reference

[noahmoss]

Modifies our doc generation scripts to include docs/agent-api as a path that gets picked up for generating documentation and referenced from the root llms.txt file. An agent-api/reference.md file is added to the Metabase repo in metabase/metabase#68914

I'm not 100% sure if this should be included in LLMS_FULL_TO_GENERATE since this is only a single static llms.txt file for the agent API feature for now. But in the future (e.g. if the API surface area expands) it may make sense to separate it out into multiple files, so maybe a special carve-out isn't needed here?

[heypoom]

I'm not 100% sure if this should be included in LLMS_FULL_TO_GENERATE since this is only a single static llms.txt file for the agent API feature for now.

This will generate only the fully concatenated https://metabase.com/docs/llms-agent-api-full.md variation of the llm file. It is fine if the agent-api/ references are not too long and in Markdown format. If there are lots of content for agent-api/, I suggest creating a https://metabase.com/docs/llms-agent-api.txt instead where it is directories of links.

Reason is that the full context usually bloats the context window, and it's not going to be able to focus as much compared to fetching only a few documents as needed.